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Introduction 



Background and Objectives 

"Developmental" projects have become an important and distinct segment of 
research-related activities in education. It appears logical to assume tliat 
the present move to give states greater autonomy in the use of research- 
rel ited funds will result in a further increase in developmental activities. 

On the other hand, very little guidance is available to potential "developers" 
to assist them in understanding the "developmental process" or in planning 
"developmental" projects. Textbooks on research methodology and programs for 
training researchers typically ignore the potential differences between the 
research and the developmental processes. While pe-^adigms are available for 
research, they have not been formulated for development. 

Since an irrportant responsibility of Research Coordination Units is to 
provide technical consultation to persons involved in research-: 'lated programs 
it seems mandatory that Unit personnel (a) attempt to understand the develop- 
mental process, and (b) begin to formulate and organize guidelines that will 
assist their clientele in planning and conducting developmental projects. 

The "Developmental Project Guidelines Conference" grew out of this con- 
cern. Since so few published materials were available, and because Unit 
personnel had limited first hand experience with developmen ca 1 projects, it 
was decided to organize an exploratory conference — one in which a small 
group of carefully selected participants would be encouraged to pool their 
relevant experiences and ideas in a relatively unstructured situation. Tt 
was felt that a more fruitful, creative approach to the investigation of the 
general problem area could be obtained by using available expertise in tl\is 
manner, rather than by utilizing the more typical but reSv^rictive technique 
of requesting reactions to materials pre-developed by the conference directors. 
The major purposes of the conference were seen as (a) helping Unit personnel 
(and through publication other concerned educators) develop their concept 
of the developmental process, and (b) providing an opportunity for participant- 
experts to learn fioni each otleT. 



Conference Organization 

oix participants were invited; they, plus a confeience leader and the three 
conference co-directors constituted the uotal number of conferees. Three of the 
participants were asked to prepare papers for circulation in advance of the con- 
ference. The other participants were invited to submit written reactions if 
they desired. During the two and one-half day meeting, each author of a paper 
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was given limited time to discuss his or her prepared material; the remainder 
of the time was devoted to free-flowing discussion. 

Since it was necessary to provide some kind of guidance to participants 
in advance of the conference, lwo major questions (with some subquestions ) o 

concern 'to Unit personnel were formulated and "^ftions 

While discussion, and the prepared papers, were not limited to these questions, 
they proved fairly central to the concerns and perceptions of the participants, 
and were therefore of some organizational value. 

The leader selected for the conference was a staff member of r.he Upper ^ 
Midwest Regional Educational Laboratory. Not only did the ^ 

the bulk of the financial support for the conference as a service to the Coor 

dination Unit, but it also is vitally concerned with the 

Laboratory supports and conducts developmental projects. Immediate, fi 

hand feedback to the Laboratory was thereby assured. 

The conference was held June 13 - 15, 1968 at the Thunderbird Motel, 
Minneapolis, Minnesota. 



The Report 

Obviously, this report is not a complete transcript of the 

fonfe^ence, a ^ery relevant paper by one of the participants originally prepared 
for publication in a journal, a summary of the discussion, and some tent ti t 
conclusions reached by the conference co-directors. The conferees co-direv.toi s 
were responsible for summarizing the discussion in the form contained in this re- 
port. 'iL participants had two opportunities _ to review the discussion summ ry; 
first while at the conference, and again in its present written orm. , 

therefore hoped that the material reflects the essence, if not the exact words, 
of the discussion. Not all the material reported was agreed upon by all the 
participants, although most of it had the support of the majority. 

No attempt was made to credit individuals with specific statements. Each 
participant played an important role throughout the meeting, lo eaci, , 
co-directors, express our sincere appreciation for their coopera ion 
stantive contribution represented by the content of this report. 
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Conference Guidelines 



Two questions of major concern were posed to give direction to, and serve as 
guidelines for the preparation of papers prior to the conference and to direct 
and guide discussion and reactions to the papers at the meetings. 



1 . What is an educational development project? . . .was posed in order to 
direct attention to the definition and description of developmental 
projects in terms of the: 



a . 



possible relationship (similarities and peculiarities) among 
developmental projects, research projects, school operations 
projects and dissemination projects regarding the initiation 
problems, outcomes, and activities of each. 
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b. implications of the above differences and similarities for 
i) planning and ii) operation of developmental projects. 



2. How should one go about planning (or condu cting) educational dg^lo 
m ent projects? ..., .was posed in order to investigate possible guidelines 
for or description of: 

a. the steps and/or sequence of action in planning a developmental 

project . 

b. the interrelationships among those steps. (Are the steps the 
same for each kind of development project?) 

c. the implications of the above procedural characteristics for 
staff, processes and organization. 



Conference Agenda 



June 13th 

9:00 - 9:30 a.m. - Opening Remarks 

Dr. Maxwell, Conference Leader 
9:30 - 10:30 a.m. - Presentation of Paper 

by Dr. Stromsdorfer 
10:30 - 11:00 a.m. - Coffee 
11:00 - 12:00 a.m. - Presentation of Paper 

by Dr. Grobman 
12:00 - 1:30 p.m. - Lunch 

1:30 - 2:30 p.m. - Presentation of Paper 

by Dr . McNeil 

2:30 - 3:00 p.m. - Coffee 

3:00 - 5:00 p.m. - Discussion 

5:00 - 7:00 p.m. - Dinner (arranged) 

7:00 - 9:00 p.m. - Discussion 

June 14th 

9:00 - 9:30 a.m. - Summary of First Session 

Dr. Moss 

9:30 - 10:30 a.m. - Presentation of Paper 

by Dr . Gideonse 

10:30 - 12:00 a.m. - Discussion 
12:00 - 1:30 p.m. - Lunch 

1:30 - 3:00 p.m. - Discussion 

3:00 - 3:45 p.m. - Coffee 

3:45 - 5:00 p.m. - Discussion 

June 15th 

9:00 - 9:30 a.m. - Summary of Preceding Sessions 

Dr. Moss 

9:30 - 12:00 a.m. - Conclusions & Recommendations 
12:00 Adjournment 
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Introduction 



Much research designed to evaluate new techniques in education is 
limited in scope and does not permit comparative judgments to be made 
between alternative methods of achieving the same objective. It is often 
research which shows what the absolute impact of a technique, say, the 
introduction of a teaching machine, is. And, of course, one can discover the 
general cost dimensions of this new technique in terms of time and resource 
outlay. However, the design and execution of the research is such that no 
comparative analysis can be made of the technique in question and all other 
alternative courses of action. Thus, while this type of research is of value, 
it is only of partial assistance in making efficient choices and, to the extent 
that such research is used in making choices, it does so only imperfectly and 
at the risk of making errors; errors in the sense that a different technique 
might have achieved the same result at a smaller expenditure of time and 
resources . 

Economic analysis of all types of investment in the human agent has come 
into vogue in recent years. Some of it has been very good, but much of it 
is badly and inexpertly done. For instance, analysis is often conducted (and 
even paid for at nontrivial prices) where one of the most elemental aspects 
of research design, an appropriate experimental and control group, is lacking. 
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Economic Analysis of Educational Alternatives 

This paper specifies some of the basic principles of prcper research 
design in the economic evaluation of developmental and other types of 
educational projects. This economic evaluation is known under different 
titles — cost-benefit analysis, cost-effectiveness analysis, systems analysis 
and so forth — but it is simply economic analysis. 

What does economic analysis have to offer in the area of educational 
decision making? It offers a frame of reference designed to systematically 
investigate the competing claims of alternative means of achieving the same 
objective. Where alternatives do not exist, economic analysis is not relevant. 

This type of economic analysis has three major characteristics. First, 
it is quantitative. As will be shown below, costs and benefits can be expressed 
in either monetary or non-monetary terms. But for the models we present 
above no non-monetary costs will be considered. Second, this analysis must 
be directly related to the objectives being served by any given set of 
alternative programs. Thus, the proper definition of the objective or 
objectives being served is critical to the analysis. Ill-conceived specifi- 
cation of objectives as well as ill-conceived choice and construction of indices 
to measure the attainment of these objectives will result in a failure of 
the analysis to provide information in making choices among competing alter- 
natives. Finally, this analysis links costs with benefits. 

Treatment of either costs or benefits in isolation of the other cannot 
provide valid information in making choices. Costs and bene-f^its of the 
entire program, however 5ts limits are defined, must be considered in conjunc- 
tion for any evaluation of alternative programs or actions. The net effec- 
tiveness of any program is due to the joint effect of costs and benefits as 
these have their impacts over time. 

These steps, then, must be followed in order to undertake an evaluation 
and comparison of alternative programs: 

1. Objectives of the program must be specified; 

2. Processes or activities to implement the program 
objectives must be developed; 

3. An index or indices of performance of the activities 
which are intended to measure programveff ectiveness 
must be specified; 

4. A production function must be specified by which the 
output of any given activity can be related to a set 
of related inputs; 

5. A cost function based on the production function 
given for each activity must be specified; and. 
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6. A comparison between benefits (performance indices) 
and costs must be performed. 

These points will be considered in turn. 



Program Objectives . Specification of program objectives is critical 
to any comparative analysis of programs. Education itself has multiple 
objectives. For instance, the objectives of education can be ones of 

a. Efficiency or increasing economic output; 

b. Equity or income redistribution; 

c. Pure consumption or enjoyment of education for 
its own sake; and, 

d. Socialization or development of socially effective 
behavior. 

The achievement of each of these objectives satisfies specific desires or 
wants of society or individuals. The particular manner in which the achieve- 
ment of these objectives is related to the creation of satisfaction or 
increased well being is described in what is known as a welfare or utility 
function. 

Each of the above objectives enters into a welfare or utility function 
for any society which engages in the production of education. Each of these 
objectives enters into the utility function of any individual who provides 
or partakes of education. 

However, the four objectives listed above are much too broad to be 
directly amenable to analysis and must be broken down, if possible, into 
empirically manageable components. For instance, the equity objective for 
education or any given developmental program in education should be specified 
as to the nature and amount of the income redistribution which is desired 
as well as the groups which are to be potentially benefited or, perhaps, 
penalized. 

Thus, welfare or utility must be defined in terms of its components, 

\«Aiich themselves must be made much more specific so that they can be measured 
in some way. Thus, if economic efficiency is one of the elements contributing 
to social or private welfare, economic efficiency must be expressed in terms 
of some measurable quantity. Then, to maximize that quantity of welfare 
gained which is attributable to efficiency, one should maximize efficiency. 

For a given developmental program in education, efficiency is maximized by 
minimizing cost (monetary and non-monetary) subject to some specified level 
of gain or by maximizing some particular benefit subject to some specified 
level of cost. 

If the efficiency objective of a developmental program is to increase 
earnings of the participants, then the maximization of the net addition to 
earnings attributable to education relative to the costs of achieving that 
education is a first step in achieving economic efficiency. Yet, again. 
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once this is done, only one aspect of utility or welfare has been maximized 
and the single-minded pursuit of this objective may result in a reduction in 
marginal benefits to be achieved from other objectives on which total wolfare 
depends. The interrelationship of these objectives must be borne in mind. 

Thus, economic analysis of education must always be partial in scope 
and, indeed, the economic analysis of any given educational activity or 
program must be partial in scope. This limitation of economic analysis must 
always be borne in mind. For instance, one must not lose sight of the fact 
that maximizing the new present value of an income stream attached to a 
person or set of persons participating in a vocational-technical curriculum 
is the same thing as maximizing the overall utility to the person or society 
resulting from the vocational-technical curriculum. And, also, it must be 
stressed that maximizing the new utility flowing from the vocational- 
technical curriculum does not necessarily imply an equal net addition to the^ 
total utility of education as a whole. There may be ways in which the pursuit 
of objectives at a lower(higher ) level in a program actively confli t with 
or contradict the pursuit of objectives at a higher( lower ) level in the 
overall context of an educational program. 



Activities to Implement Objectives ■> Herein lies the development of 
alternative activities to pursue the objectives of the program. These 
activities can encompass two or more different ways of pursuing the same 
activity in order to achieve a given program objective. Or, they can involve 
two or more different activities to achieve the given program objective. 

In the first case, if the program objective is, for instance, to maximize 
the present value of net earnings, one may develop different curricula in 
order to see which curriculum, for a given cost, yields maximum net earnings. 
Comparisons could be made between college preparatory, general and vocational 
technical curricula, for instance, to attempt to discover which among a set 
of instructional techniques imparted the maximum net addition to one's know- 
ledge of mathematics most efficiently imparted— by the teacher in person, 
by teaching machines, or by television instruction? Or, what is the optimum 
size of class, other things equal, for maximizing performance on some 
standard performance test or set of tests. 



Specification of the Pe rformance Index . Given the activities chosen are 
appropriate to pursuing the objectives of the program, one of the most 
critical states in the analysis is to develop an index or set of indices to 
measure performance. Any such index can only be an approximate measure of 
the output of the activity. What, for instance, is the output of a senior 
high school? Is it the number of graduates? Is it the creation of educated 
persons? What are educated persons? What is education? Education is a 
process and an output. What is the nature of education as an output. Is 
it a stock of knowledge? Is it the ability to reason? Is it the ability 
to recognize and appreciate the "Good"? 

Clearly, the definition of the output is crucial, for a program or 
activity must be organized and defined in terms of its output. 
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To continue, what is an index for measuring this output? If the output 
of education is the acquisition of a store of knowledge and the ability to 
reason, then performance on a standard test to measure these^ two ^ components 
of educational output may oo an appropriate index. But, again, it will be 
an approximate measure since it will never be conceptually nor practically^ 
flawless. For instance, no test can measure all aspects^of reasoning ability 
or knowledge. No test can distinguish between that ability to reason which 
is innate and that which is developed through the learning process. Finally, 
no test can measure ability to reason independent of one's stock of knowledge 
because a basic store of knowledge is needed as a frame of reference for 
all reasoning. Thus it goes. It is net necessary to multiply further examples 
to demonstrate the complexity of this problem. 



The Production Function . The production function specifies the physical 
output, or outputs, the physical inputs, and the relationship between input 
and outputs, and, in some cases, interrelationships between subsets of inputs. 
This function should show the various combinations of inputs which can be 
used to produce a given level of output. It should also show how output 
increases or decreases as the proportion of inputs changes. 

The general form of a production function is as follows: 

(1) X = f(a,b,c,d) where X is output and 
a,b,c,d are inputs. 

If the number of students graduating defines the output of a senior 
high school, then the inputs could be the number of teachers of a given 
quality and type, number of classrooms, kilowatt hours of electricity, number 
of administrative staff, amounts of different supplies and so forth. Finally, 
this flow of inputs which creates a flow of outputs must be related to time. 



Cost Fu nction . It should be possible to measure each of these inputs of 
the production function in terms of its money cost. The production function 
of a secondary school or of any educational activity and the prices of inpu s 
determine the cost function. For this cost function it is possible to 
estimate a total cost of any given activity and also a marginal cost, that 
is, the extra cost of producing an extra unit of output. Thus, if output 
is defined as an index of performance on a standard test, and if the unit 
of observation is that student who has achieved a standard level of perform- 
ance on the given test, then the total costs of achieving that level ^ of 
performance can be estimated for a given number of students. And, given 
that costs are related to the student, the marginal (extra ) cost of training 
an additional (extra) student at that level of proficiency can be estimated. 

Other relationships which are not cost relationships can also be 
estimated. For instance, test performance or the probability graduation 
can be expressed as a function of expenditure per student as well as student 
characteristics, such as teacher quality, size of school, school location 
and other variables. In such an estimated relationship ^ 

a cost function) one can calculate the net contribution of these educational 
inputs (as these are measured in dollars) to test performance or the pr ^ 
ility of graduation. Of course, the contribution of each these inputs is 
net only in terms of the other elements expressed in the estimated relationship, 
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Relation of Costs to Benefits . If benefits are non-monetary in nature, 
then a target level of program performance can be set and that activity w^ich 
achieves the performance target at the lowest cost (both monetary and non- 
monetary) is the desirable program. Or, a given cost can be set, and that 
program which achieves the highest performance level is the desirable program. 
If both benefits and costs are in monetary terms then internal rates of 
return to the activities used to implement program objectives can be easily 
calculated. That activity with the highest internal rate of return is the 
desirable activity. However, to avoid the possible error that the absolute 
size of gain and cost will be ignored, net present values of benefits should 
also be estimated. That activity with the highest net present value is the 
desirable activity for achieving program objectives. There are serious 
theoretical difficulties in the practical estimation and application of the 
internal rate of return and net present value decision rules but these are 
beyond the scope of this paper. 



Generality of the Framework . This simple framework for evaluating an 
educational activity or project is completely general. Given that objectives 
are clearly specified, alternative projects to achieve these objectives can 
be established. Input combinations between alternative projects will 
likely vary. Input combinations within a given project can be varied. 

The effects of these two types of variation can be noted on both output 
and on input costs. The combination of inputs for a given cost, which will 
then maximize a given output, can then be discovered. 



A Simple Model i The Neighborhood Youth Corps . The foregoing can best 
be expressed in terms of setting up a simple model to evaluate the effect- 
iveness of a particular social investment project in the area of education. 

The Neighborhood Youth Corps (NYC) was established in part as a dropout 
prevention program. Its intent is to either return dropouts to school 
(the Out-of -school program) or to prevent students from dropping out (the 
In-school-and Summer programs). 

One stated goal of the NYC is, then, to encourage students to continue 
school who otherwise would have dropped out. Consider the Summer NYC program. 
An index of its output would be the following: For a given cohort of students 

who complete a school year and who also complete the Summer NYC program, what 

is the probability: 

a. That they will enter the fall term of the next 
school year; or 

b. That, having entered, they will complete the fall 
term of the next school year; or 

c. That, having entered, they will complete the 
entire subsequent year? 

Any of these three measures could be an index of output for the Summer 
NYC program, depending upon the rigor which one would want to impose as a 
measure of the ’’success” of the Summer NYC program. 
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In order to measure the net effect of the Summer NYC program we want 
to know how many students who would have dropped out subsequently continue^ 
their education only because of their participation in the program. Techniques 
to measure this directly do not exist. We do not know who the^ potential drop- 
outs are before the fact. And many students will participate in the program 
who will never dropout in any case. We cannot appeal to simple comparisons 
in gross dropout rates before the program and gross dropout rates aft er the 
program. Any number of intervening influences could have occurred over the 

summer to affect dropout rates. 



since "before-after" comparisons are usually irrelevant or erroneous, 
a control group from the same cohort of students as those who enroll in the 
Summer NYC program will need to be chosen. Then in a properly specified 
empirical model, the net effect of the Summer NYC program can be ted. 

The simplest model would relate the probability of continued enrollment or 
continued education to the state of having participated in the Summer NYC 
program. Thus, 



(2) Y = f(X) 

where Y equals the status of enrollment or continued education and X is the 
status of the student with respect to the Summer NYC program. More speci 

ically, 



(3) Y = a + biX^. 

where Y takes on two values as follows: 



1 if the person continues his education; and, 
0 if he does not. 



takes on two values: 

1 if the person was a Summer NYC participant; and, 

0 if he was not; and, 
a and b^^ are parameters. 

The empirical estimation of this model might yield the following results 



(3a) Yi ~ .53 +• .10 X^ 

(.21)*(.03) 

One could then say that, other things equaj,, 

Summer NYC program resulted in a ten per cent increase in the 

that a student would continue his education in --iiy related 

should be expanded to include other variables which are theoretics y 
to the probability that one will continue his education so as to ge 
preciL measurement of the contribution of the Summer NYC program to probab- 

ility of continuation in education. 



*The numbers in parenthesis are the hypothetical standard errors 
hypothetical partial regression coefficients. 
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One variation of this model v«)uld be to include the socio-demographic 
characteristics of the persons in the control and experimental (NYC) groups 
into the model as well as school associated characteristics. In addition to 
this, one could include the expenditure per NYC participant into the model. 

Then, the partial regression coefficient of this expenditure variable would 
indicate the net contribution of a given amount of expenditure to increasing 
the probability of continuing education. Thus, if 

(4) Y 5 + bgXa + bsX 3 

v^ere X 2 stands fo^' a set of socio-demographic characteristics and X 3 represents 
expenditure in units of ten dollars and the other variables are defined as 
in equation (3) above. The estimated relation then might be 

(4a) Y * 



In this hypothetical case, for each ten dollar expenditure on a student 
in the Summer NYC program, the probability that one would continue his 
education would increase three-tenths of one per cent. 

This relationship, of course, is not a cost function. The estimation 
of a cost function for the Sumner NYC program would be as follows 1 



where Y equals the total cost of a given Suraner NYC program based on pricing 
out the inputs of the program's production function, Xj^ equals enrollment in 
the program and a and hi are parameters. The partial regression coefficient 
bx, would display the marginal cost of enrolling an extra participant in 
the program. This is a linear function as expressed. Since the total cost 
function may not be linear, other functional forms may be estimated to arrive 
at the most correct form, given the data, the program, and the underlying 
economic theory. 

There may be several different approaches to achieving the objective 
of the Summer NYC program. Each of these will have its own production function 
and its own cost function. If the success of each of these alternative tech- 
niques in promoting the continuation of education is equal, that technique 
which has the lowest marginal cost will be the most efficient one to use. 

It will be more efficient (or effective) in the sense that a given level of 
attainment of the project objective can be obtained at the lowest cost outlay. 

If the alternative techniques have both different costs and different 
degrees of success in fostering the continuation of education, then the diff- 
erential failures (the dropouts) can be treated as production rejects, implying 
that one technique is more costly or causes more "waste" than another. Then, 
as long as the socio-demographic characteristics of the participants as well 
as school characteristics for the different techniques are the same, one can 
still make a proper choice of the most effective program. 




(5) Y = a f biXi 
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However, it is certainly not likely that the school characteristics and 
socio-demographic characteristics of the program participants will be the 
same among the alternative techniques being evaluated. In such a situation, 
multiple regression and correlation analysis can provide information, based 
on an appropriate theoretical model, to identify those variables which have 
the greatest relative net effect on the output variable. Sub-sets of program 
participants can then be developed based on this analysis. For instance, it 
may turn out that race, sex and IQ have the greatest relative impacts on prob- 
ability of continuing one's education. Then, among the alternative techniques 
of the Summer NYC program, the marginal costs of white males with IQ's falling 
in a range from 100 to 110 could be compared between alternative techniques. 
This could likewise be the case for other possible combinations of personal 
characteristics among the NYC participants. One might then discover that one 
technique is more effective (the marginal cost per continuing student is less 
for one technique) given one type of participant (say, white females with IQ's 
of 100 to no) but that it is less effective for another type of participant 
(say, Negro males with IQ's greater than lOO). Finally, if efforts to stand- 
ardize the characteristics of the participants are not made, then one will 
not be able to tell whether the differential marginal costs between techniques 
are due to the different production functions of the techniques, the different 
characteristics of the participants, or interacting combination of both. 

The analysis will be inconclusive. 



Examples of Empirical Analysis . A variety of economic and statistical 
analysis of educational programs has been performed. Two such studies are 
considered here. High School Size and Cost Factors ^ and Equality of 
Educational Opportunity^ . 



The study by Herbert Kies ling is of interest because the analytical 
model he uses in the study is similar to one of those suggested below. He 
postulates the following model 



■ a ♦ ^ ^2^2 ^ ^3^3 ^ ^ 4^4 ^ ® 

where 

Yi is the measure of quality 

= verbal Knowledge of Factor Score (intelligence) 
X 2 = expenditure-per pupil in ADA 



1. Herbert J. Kiesling, High School Size and Cost Factors , Final Report, 
Project No. 6-1590, U.S. Office of Education, March, 1968. 

2. James S. Coleman, et alia . Equality of Educational Opportunity , 

Washington, D.C.: U.S.G.P.O., 1966. 

3. Kiesling, 0 £# cit . , p. 33. 




- high school size in ADA 

X4 - average value of the socio-economic index for 
pupils in the population for which Y is 
applicable. 

e - an error term 



Kiesling argues that he is attempting to measure school quality. From 
a decision making standpoint j an attempt to measure "quality" by itself^ is 
not too meaningful. Given that extra expenditure will yield extra quality, 
the question still remains as to how much extra quality to incorporate in a 
program, or which program of a given quality to adopt. 

However, to continue, his measures of "school quality" are of three 
types. One of these three quality measures are multiple choice question 
tests constructed along subject lines which include English, Mathematics , ^ 
General School (academic subject) Aptitude and General Technical Aptitude. 

In simple terms, the relationship between his quality variable and his expend 
iture variable may look as indicated in Figure 1 . Here, he is actually 
dealing with an index of output, test performance, as well as an index of 
Quality. What is shown here is that as expenditure increases, test performance 
increases at a decreasing rate. A way of looking at the quality problem more 
consistent with an economist's viewpoint is to compare relative test perform- 
ance of pupils being taugh?. by different techniques but at the same marginal 
cost outlay for each technique. Then, that technique having the highest 
relstive score can be said to be more efficient or effective. 



Figure 1 : Hypothetical Relationship between and X2> 

Net of the Influence of X^, X^ and X4» 




4 . Ibid. , p . 8 . 
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In short, in the strictest terms his study shows the net change in test 
performance for a change of a given unit of expenditure but this doesn t 
tell one how efficient a particular school is in teaching that subject to a 
student. And of course, expenditure itself is not a measure of quality, since 
different expenditure levels may simply represent different combinations of 
teaching inputs among alternative techniques while saying nothing about the 
resulting output levels among techniques. To summarize the findings of 
Kies ling, the estimated relationship between test performance and expenditure 
appears in general to be linear and upward sloping though in some cases the 
new relationship is similar to that expressed in Figure 1 below. Finally, 
he states that^ 

After allowing for these control variables (X^, X 3 , and X 4 
above), it was often found. . .that an additional $100 of 
expenditure per pupil (in ADA) was associated with between 
,1 and .2 of a standard deviation in the dependent variable. 

This is no small effect. 

What does this $100 of expenditure represent? It represents the average 
input mix of instructional inputs for either the sample as a whole or for 
the sample subsets for which he estimates relationships. The reader does 
not know what this input mix is. And, he has no information on how changes 
in the input mix that can be bought for $100 will affect performance scores. 
Thus, his statement "This is no small effect" does not have any meaning. 

Small relative to what? 

Of course, Kiesling's study is not a cost-effectiveness study, so that 
this criticism is to an extent unfair. But the criticism is not unfair to 
the extent that Kiesling purports to be discussing educational quality. 

In contrast, the Coleman Report ( Equality of Edu cational Opportunity), 
which is based on the same project talent data which Kiesling uses, findsP 



. . .that social composition of the student body is more 
highly related to achievement, independently of the student's 
own social background, than is any school factor. 

Further, the findings can be summarized as expressing the phenomenon 
that' 



5. Ibid ., p. 132. 

6. Coleman, et ££. > P" 325. 

7. Samuel Bowles and Henry Levin, "The Determinants of Scholastic Achievement^ 
An Appraisal of Some Recent Evidence," The Journal of Human Resources, 
Winter, 1968, p. 4. See also Coleman's reply to this critique in the 
Spring 1968 issue of the same Journal. Also to be published by Bowles 

and Levin in the Summer 1968 issue of the same Journal is the xx-, Equality 
of Educational Opportunity: More on Multicollinearity and the Effective- 

ness of Schools." Forthcoming. 
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per pupil expenditures, books in the library and a number 
of other facilities and curricular measures show very little 
relation to achievement if the social background and attitudes 
of individual students and their schoolmates are held constant. 



Thus, while different subsets of the same data set were used, the two 
authors come to contradictory conclusions. The difference is, in part, 
due to the fact that the Coleman report includes a considerably larger 
number of independent variable in the estimated equations. But, perhaps 
the difference is more apparent than real, since as Bowles and Levin point 
out, the Coleman Report finds a significant relation between achievement 
and teacher's characteristics. And teacher's characteristics account for 
75 per cent of the variation in teacher's salaries. And teacher's salaries 
dominate the institutional expenditures category.® 



Also, Kielsing used a dummy form for his expenditure variable, breaking 
expenditure into $50 ranges. The Coleman Report used a linear continuous 
variaLle form. There is some evidence that the empirical functional relation- 
ship between performance and expenditure is not linear. Economic theory would 
support this non-linear relationship on a priori grounds also. Thus, if 
Kies ling's formulation more closely approximates the true functional relation- 
ship, which it probably does, his estimations will have a higher degree of 
statistical significance. 



Also, the manner in which the regression analysis was conducted imparted 
a bias to the findings on the relation between achievement and expenditure 
in the Coleman Report. The technique in the Coleman Report was to add each 
independent variable in a stepv/ise fashion and then display the difference 
between the coefficient of multiple determination for the equation with a 
given variable in it and the coefficient of multiple determination for ^ the 
same equation but excluding the variable in question. This procedure is only 
valid if the set of independent variables are completely^ independent of each 
other. But such is not the case for the variables in this study. School 
characteristics, student characteristics, and expenditure levels are all 
intercorrelated. Thus, the order in which variables are introduced into 
the equation will affect the observed difference in the coefficients of 
determination. It is then possible to structure the order of independent 
variables so that, say, variable two which is added after variable one, 
but which is highly correlated with it, adds little or nothing to the 
explanatory value of the overall relationship. Finally, the Coleman Report 
does not display the partial regression coefficients so that one cannot ^ 
determine the amount by which a unit of expenditure effect test performance. 

Neither of these two studies provides the data analysis necessary to make 
choices among competing educational alternatives, though the Coleman Report 
presents information which would tempt one (erroneously) to make economic 
' judgments. Of the two, the Kiesling study appears to be more consistent with 

the needs of economic analysis. 



8. Bowles and Levin, "The Determinants of Scholastic Achievements," 0 £. 
cit . , p. 10. 
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Summary . This paper presents a general framework for evaluating 
developmental and other types of educational projects. The framework is 
presented in the context of economic analysis, sometimes known as systems 
or cost-effectiveness analysis. It can deal with efficiency problems 
concerning both economic and non-economic outputs. It can be applied, and 
should be, to any situation where choices among alternatives are being made 



ERIC 



Selected Bibliography : 



1. Gary S. Becker, Human Capital , (New York: Columbia University Press), U 

1964. 

2. Samuel Bowles and Henry Levin, "The Determinants of Scholastic 
Achievement — An Appraisal of Some Recent Evidence," The Journal of 
Human Resources , Winter, 1968. 

3. , "Equality of Educational Opportunity: More on Multicoll- 

inearity and the Effectiveness of Schools", The Journal of Human 
Resources , Summer, 1968. Forthcoming. 

4. Mary Jean Bowman, et alia . Editors, Readings in the Economics of Educa- 
tion , (Paris: UNESCO), 1968. 

5. James S. Coleman, ^ alia . Equality of Educational Opportunity , (Washington: 
USGPO), 1966. 

6. Herbert J. Kiesling, High School Size and Cost Factors , Final Report, 

Project No. 6-1590, U.S. Office of Education, March, 1968. ^ 

7. "Investment in Human Beings," The Journal of Political Economy , 

Supplement , October, 1962. 

8. Roland N. McKean, Efficiency in Government Through Systems Ana lysis, 

(New York: John Wiley and Sons, Inc . ) , 1958. 

9. Roland N. McKean and James A. Kershaw, Systems Analysis and Edu cation, 
RM-2473FF, RAND Corporation, Santa Monica, Calif., October 30, 1959. 

10. Burton A. Weisbrod, "Conceptual Issues in Eyaluating Training Programs." 
Monthly Labor Reyiew , October, 1966. 



- 21 - 



The Economic Evaluation of 
Development Projects in Education 



Reaction To The Stromsdorfer Paper 

by George Maccia 



Professor Stromsdorfer has pointed to the limitations of use of economir 
theory in educational development programs. He also indicated the necessity 
for having knowledge of the net effect of developmental programs on the persons 
involved. To obtain such knowledge we need to know which students are affected 
by them. Professor Stromsdorfer asserted that techniques to measure this net 
effect directly do not exist. Thus, he offers an indirect approach by way of 
an input-output systems analysis in which techniques of cost-benefit analysis 
are used, with regression equations as instruments of measurement, and economif. 
efficiency as the criterion of programmatic achievement. As we are all we] 1 
aware, regression equations never tell us about any ostensibly present individ- 
ual. What we get from them are estimates of the likelihood of some indefinite 
individual who is more or less likely to be represented in a defined class of 
ostensibly present individuals. 

I cannot, of course, comment on the adequacy of Professor Stromsdor f er ' s 
model as a model for analysis of economic programs. I am troubled, however, by 
an implicit assumption that educational phenomena are generally, if not totally, 
reducible to economic phenomena, with the instruments of one transferable to 
the other. He rightly rejects the pretest-postest form of analysis, once so 
popular in education, as being of little worth. He proposed group comparison 
analysis, but does not warn us of the legions of difficulties in this approacn. 
He enjoins us to tie our analyses to specified performance objectives and to 
evaluate educational development programs after meticulously fastening our 
operations to those objectives. We have often done this and h.ave found t'nat 
interaction variation in system functions which are not specifiable through 
the selected objectives could not be discerned while it was happening. Ihe 
approach (utilized in most AID programs) has been a model of disaster. This 
form of systems analysis is subject to the "thalidomide mistake". It cannot 
be otherwise, for such "side effects" are not covered either by the objectives 
selected or through* regression equations comparing inputs to outputs. What we 
do need is a general strategy for educational development which has available 
an adequate descriptive and explanatory educational theory. Through such a 
strategy and theory it would be possible to discern "side effects" arising 
from a decision to optimize specified educational outcomes. Lacking such a 
strategy and theory, we might do well to abandon group analysis and seek that 
instrument which can predict whether Johnny will "drop-out" or continue in a 
given educational program. Despite what Professor Stromsdorfer avers, there 
are instruments available for identifying the potential drop-out Johnnies. 

These are teachers with long experience with students from given socio- 
economic groups. 
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Reckless and Dinitz* recently uncovered the research and developmental value 
of such instruments. They were not interested in the potential drop-out, 
however, but in potential delinquents, as defined by the courts in Columbus, 
Ohio. In trying to predict which students in a given population would be 
judged by the courts as delinquent five years from the time of their measures, 
they developed regression equations utilizing socio-economic factors and 
test results. They also asked teachers with ten years experience in teach- 
ing such students who they thought were the potential drop-outs. The best 
estimate yielded from the application of regression equations was approxi- 
mately 70^. The teachers' accuracy ranged from 90 to 9 ^%, Perhaps if we 
ask teachers the right questions,' calling for predictions of Johnny's be- 
havior, rather than reasons for Johnny's behavior, we may have that instru- 
ment which will yield predictions which enable us to devise techniques for 
ascertaining the net affect of developmental programs of education. If we 
selected our best such predictors we then could more readily interpret ec- 
onomic efficiency in terms of Johnny, rather than Johnny in terms of econo- 
mic efficiency. 



Discussion 

Following the opening remarks and orientation to the conference purposes 
by the conference leader. Dr. Maxwell, Dr. Stromsdorfer began the presentation 
of his paper. The discussion which it generated prevented him from getting 
very far into his paper. This discussion included the following points, topics 
and questions. 

1. The "efficiency" and "effectiveness" of developmental projects in 
education were defined. "Efficiency" is the program's ability to optimize 

a specified utility function. Systems analysis takes into account all measur- 
able costs and expenses in terms of monetary units. If monetary values can 
be assigned to outcomes, efficiency can also be expressed as investment re- 
turns. At present, economic analysis is limited by our ability to place a 
monetary value on all pertinent educational costs and outcomes. "Effective- 
ness" of a program is the extent to which it achieves all of its relevant 
outcomes, both economic and non-economic . Cost-effectiveness analysis is 
employed as an evaluative tool when either costs or benefits cannot be ex- 
pressed in purely monetary terms. 

2. Tentative probes were made at defining "developmental" projects. 

The trilogy of research, development and operation was advanced as a useful 
classification of educational activities. (Dissemination was also mentioned 
as a separate form of activity at a later point in the discussion.) 

3. The output of research was postulated to be knowledge, while the out- 
come of development was thought to be a process or product immediately ap- 
plicable to the school situation. 



*In In terdisciplinary Problems in American Criminology . Papers of the 
American Society of Criminology, 1964. 

For a complete analysis of this work see additional papers referenced 
in Criminal Law and Criminology 55:515-523, 1967. 
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4. Development involves engineering or designing processes or products 
that can be installed in an operating system to accomplish the ends of that 
system. Development utilizes the results of prior research or practice by 
applying knowledge in new ways or in new combinations. It is a process for 
effectively operationalizing our educational knowledge. 

5. It was felt that there is an inevitable relationship and even over- 
lap between research, development, and operation. 

6. fhe role and potential constraints of setting project objectives 
early :,n the developmental process were discussed. The complexity of the 
developmental product, as peiceived by the developer, is instrumental in 
determining the degree of specificty with which he can, or is willing to, 
state expected outcomes at the start of the project. 

7. No consensus among the conference participants was reached regard- 
ing the above points. 
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A summary identification of decision points and alternatives in developtnental 
curricula.^ 



While there is a multitude of differences among the various developmental 
curriculum projects^ that have been functioning at some time during the last 
10 years, there are many common decision points. Perhaps ways in which projects 
go about handling these decisions and the concomitants^ of the possible alter- 
native courses of action provide a framework for describing the developmental 
curriculum project movement as it presently exists and for identifying^the most 
promising ffiture directions, (it should be noted that the decision points are 
not always clearly identified by those making the decisions, and the alternatives 
available and their, ramifications are often not clearly recognized by "the 
decision makers prior to -or, often, even after- the decision has been made.; 



Attention to the decision points and alternatives is based on the ^assumption, 
not universally held, that an awareness of alternatives improves decision-making, 
and that deliberate decision-making is preferable to intuitive operations, 
free from conscious concern with the making of decisions and the systematic ex- 
ploration of alternatives. While certainly it is possible to push deliberate- 
ness in decision-making beyond the point of diminishing returns, rejection of 
efforts to improve decision-making through analysis may^be tantamount to quest- 
ioning attempts to improve teaching through the analysis of teaching, based on 
the argument that good teachers are born rather than made. Without substantiat- 
ing evidence that intuitive decision-making is superior^to deliberace decision- 
making and that there are enough gifted intuitive decision-makers to fill the 
needs of developmental projects, a clearer understanding of the decisions and 
alternatives open to developmental projects would seem indicated. 



Project decisions can be grouped in a variety of ways; for example, in 
terms of who makes the decision, the strategy or method of decision-making, the 
extent to which the decision delimits or circumscribes later decisions, the^ 
importance of its impact, or the point in the project's life that the decision 
is made. The choice among systems for classifying decisions perhaps should 
reflect the interests and needs of those using it, since one system may be more 
appropriate for a given purpose than another. In a sense, any such system is 
arbitrary, since it would be difficult to think of any but the most trivial 
decision that does not affect -and, in turn, is not affected by- other decisio. s 
and other categories of decisions; furthermore, at different points in the life 
of a project, the alternatives for a given decision may change. For the present 
paper, the following groupings will be used: 



1 Ideas in this paper are developed in greater detail in Hulda Grobman, ^ 

Decisions and Processes of Development al Curriculum Projecj^ (Forthcoming). 

2. For the purposes of this paper, developmental curricula will include 
efforts similar to those of various groups over the last decade, to 
new curricula through group activity, and with trial use of experimental ed- 
itions of materials during the developmental period, and feedback 
use to improvement of materials. For a more complete definition, see Hul ^ 
Grobman, gvai„.tnnn of Corrlculum Projects . AERA Monograph Senes on Curriculum 

Evaluation No. 2 (Chicago: Rand McNally, 1968) 
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Situational considerations 
Administrative arrangements 
Source of funding 
Structure of the organization 
Personnel involved 
Politics of the organization 

Life span of the project and/or parent organization 
Project housekeeping 
Curriculum decisions 

Purpose of the developmental curriculum 
Parameters of the project's job, nature of materials 
produced and target audience 
Basic educational philosophy, psychology of learning, 

and assumptions underlying curriculum and organization 
Preparation of materials - process and timetable 
Format of the curriculum 
Method of release of materials 
Evaluation of the curriculum 
Training for use of the curriculum 
Implementation of materials following completion 

Within this system, the categories are not discrete and the order of categories 
is not necessarily sequential. Thus, the system is intended as a handy framework 
for analysis rather than as a fixed taxonomic structure. "What are the variables?" 
and "What are the Concomitants?" are different questions from "What are the 
earmarks of a successful developmental project?" and "Which projects are more 
successful?" It is the former questions that will be dealt with here. No 
attempts is made to select a "right" or "best" decision or set of procedures, 
and perhaps no single approach is optimal. However, simply the clarification 
of decisions and variables in the situation may be useful as a starting point 
for discussing optimal strategies. 

The author is not attempting a complete census of alternatives, but hopefully 
a large proportion of the more important variables are included. For each 
section, there is generally a brief listing of some of the variables, and a few 
exploratory notes and ideas. These notes are intended only as indication of 
some ramifications, since time and space limitations preclude a more complete 
consideration here. Sections are not mutually exclusive and there is some 
duplication of items among sections. Some of the sections are more fully dev- 
eloped than are others, not necessarily because of varying complexity or 
importance, but perhaps because the author has clarified her thinking more in 
some areas than in others. 



SITUATIONAL CONSIDERATIONS 



Public opinion, political tenor of the country, and various temporal factors 
are influential in project activity; since these vary from time to time, the 
greater success of a given venture may be a factor of timeliness rather than 
other variables in the situation. For example, after Sputnik, large scale funding 
was available only for science and mathematics projects at the senior high 
school level. Today, projects concerning the disadvantaged perhaps have the 
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best opportunity for major funding. Further, the purposes and procedures of a 
developmental curriculum project may be quite different when it is among the 
first such projects ever tried (e.g. PSSC, BSCS, UICSM, SMSG) or when some 
relatively successful patterns have already been established. Another important 
variable is whether it is the first venture in a subject area for a given target 
audience; thus, the situation of Harvard Project Physics, in preparing a senior 
high school physics course, is far different than it would be had not PSSC preceded 
it. 



ADMINISTRATIVE ARRANGEMENTS 



Source of funding 

Public (OE, NSF, state, local school system)/private (foundation, publisher, 
business, other) 

Duration of grant or funding - short term/long term 

Structure of budget - flexible/rigid 

Timing of grants and overlap of grant period with organization's fiscal 
year, and with school year 

Expectation for continuation of funding and level of continuing funding 
beyond individual grant period 



The restrictions and ramifications of funding through different sources are 
generally not recognized by those who have not had direct experience with one or 
more funding sources. The funder's guidelines for programs, the delays in 
awarding grants, the cliff-hanging while awaiting grants, the time and difficulty 
in shifting funds from one function to another, the lack of guarantee of con- 
tinued financing beyond the individual grant period, the length of grant periods 
and the time required in proposal waiting and obtaining grants very clearly 
modify what is possible, may preclude certain types of operations and may, in 
fact, dictate the kinds of materials produced even when the funder has no such 
desire. 



Structure of the organization 

General or multipurpose organization (university, R and D center. Regional 
Laboratory) / specialized-purpose group (to develop one curriculum). 
Multipurpose groups include: various curriculum jobs in one subject 

area; various curriculum jobs in various subject areas; curriculum 
and other work in one subject area; curriculum and other work in 
various subject areas. 

Is initiator and developer the same person 

Who decided purpose of project 
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Who makes value judgments 

Power structure of project - who makes what decisions 

Communications - within project/with outside 

Formal and informal penalties and rewards for bad and good decisions 

History and tradition of project (e.g. pioneering/stolid) - who started the 
project and why 

Image projected by project ("messing around", "creativity" , "Piaget" , 
"inquiry" ) 

Geographic area - local/state/regional/national - centralized/decentralized 

Degree of control by a parent organization or outside group (e.g. profess- 
ional organization, university, etc.) 

Age of organization and point in the life of the developmental project 

Degree, nature and frequency of quality control interventions 



Projects vary tremendously in terms of organization charts, from the centralized 
National Curriculum Projects to the decentralized Developmental Economics 
Education Project, which involves a national, umbrella organization, but only 
very loose control and some central financing and complete decentralization of 
developmental work. Organization as indicated by organization charts is only 
one dimension of organizational differences, since charts may not reflect actual 
power and communication channels which may be the major determinants of project 
success. (National curriculum projects or Regional Laboratories with identical 
organizational charts may differ widely in terms of actual day-to-day communi- 
cation and decision-making.) These in turn may reflect the personality of the 
project director. 

How does organizational structure affect quality control? (To what extent is 
quality control desirable - problem of maximum return on investment vs. desire 
for experimentation and diversity.) Is planned change and dynamic innovation 
more likely with some types of organizational structure than with others? Some 
observers (e.g. Bennis, Changing Organizations. ) feel that the traditional chain- 
of-command type of operation is incompatible with directed dynamic innova 
yet some projects operate on a clear-cut, chain-of-command basis, and the school 
systems for which curriculum materials are intended generally do. 



Personnel involved 

Selection and personality of the director - did he originate project - is 
it "his" project 

In-house preparation of materials/subcontracting out 
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Permanency of staff - permanent, regular, year-round staff (full or part- 
time on project) / special short-term (full-time or parttime) staff/ 
consultant staff/graduate student staff 

Selection and assignment of staff - by parent organization/project/target 
audience 

Permanency of regular staff - rotating or relatively permanent 

Nature of staff assignments - specialized/flexible 

Background and qualifications of staff - subject matter experts/teachers 

Background and qualifications of those developing materials- subject matter 
specialists/grade-level teachers/science writers/students at grade 
level 

Nature of personnel - research types/educators/administrative types 

Location of staff - centralized (in one building/several buildings in same 
city)/ several different cities 

Morale of staff - breaking new ground/simply another job 

Conditions of employment - tenure, fringe benefits, professional status 

Size of staff 



The kinds of personnel and many of the conditions of employment are closely 
related to the structure of the organization, its permanency and the relationship 
with a parent organization. For example, in projects in universities questions 
arise of joint appointments with departments, and whether departmental appoint- 
ments, promotions and salaries are consistent with those of the project. In 
Regional Laboratories, given the drastic shifts of activities, is flexibility 
more important than specific competencies, and what is the effect of this on 
the project? Will highly qualified persons work without a university connection 
for long periods of time? What conditions of employment (tenure, for example, 
in a project dependent on outside funding) will insure the most appropriate 
talent for the project? How does geographical location (e.g. summer writing 
conference in Colorado or Cape Cod) influence availability of staff? 



Should the project be virtually synonymous with the Director or should it have 
an independent identity? Should the directorship change over time? What does 
this do to the project? To the image of the project? To its ability to staff 
its activities? And to the acceptability of its project? 



Projects vary tremendously in terms of size of staff working on a given develop- 
mental project, from some 70-80 at a given time to less than half-dozen at a 
time. The volume and quality are not necessarily directly correlated with 
numbers. What are the advantages of large and small groups? When is the 
larger group justified? When is it more effective? 
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Politics of the organization - external and internal 
What is the external control over the project 

What are the external influences (White House, Congress, OE, financial 
backers, profession, public school systems, state departments of 
education) 

What is the power structure of the target audiences 
What is the power structure of the developmental project 
What personalities are involved in decisions 
What compromises must be made, with whom and why 
To what extent are decisions payoffs for other favors 
Are failures permitted 

What are the communication networks within the organization, with outside 
organizations and with the target audience 

Who makes what decisions (handed down decisions/pseudo-sharing/shared) 

General project morale (including past successes and failures) 

' Point in life of the project (pioneering/old, dying project) 

Personality of power ‘ figures 

These are closely linked to the "Organizational structure" and the "Personnel" 
areas. It is included separately to emphasize the importance of values, attitudes 
and pressures in determining structure and personnel decisions, as well as 
decisions in other areas, even though such decisions are often justified on other 

bases. 

Life span of a project and/or organization 

Uni-purpose organization/multipurpose organization including specific 
developmental curriculum work 

Broad purpose/limited, specific purpose 

Deliberate decision to continue/deliberate project internal decision to be 
a one-time effort/outside decision to cut off project 

Greater efficiency of continuing projects and organizations/grea*: r creativ- 
ity of new projects 

When should a project die and who should make this decision 
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The life of a developmental project may be decided at the initiation of a 
project to be a relatively short-term one (e.g. preparation of specific mater- 
ials in one subject area for one or a few grades), a continuing one, or no 
deliberate decision may be made. For many projects, initially there was no 
clear expectation concerning the duration of the project. Other projects have 
changed from relatively short term ones to longer ones as work progressed and 
the project self-image changed. Some projects - such as those in regional 
laboratories or curriculum laboratories - are part of continuing organizations 
which have as a major purpose the continuing development of curriculum. 

The question of project life span involves some basic issues in terms of 
efficiency, effectiveness and long-range educational desirability. When a 
project has once prepared a curriculum or carried out some developmental 
function, what are the gains in efficiency in going to another similar job? 

Over time, do some projects become more efficient and/or 'effective than others? 
Should this be a criterion for continuing? Are continuing projects more likely 
to institutionalize earlier innovations or arrangements so that they cease to 
be highly creative? Can an established project be as innovative once it has 
developed a successful formula for curriculum materials? Can long-term projects 
continue to attract the high quality personnel new projects have attracted in 
the past? What rewards are needed to maintain staff quality over long periods 
of time, particularly where the project does not have close professional ties, 
as with a university or professional association? 



Project housekeeping 

While adequacy of housekeeping details does not make a project excellent, 
lack of concern with detail can burden or delimit a project. Some projects 
operate on a businesslike basis, providing a level of accommodations and manager' 
ial services to staff, visitors, teachers associated with tryouts that are not 
normally associated with educational projects. This is not a matter of luxury 
spending, but rather includes such items as adequate secretarial service, fac- 
ilitating transportation, freeing professionals from non-professional tasks 
and paying people for time spent above normal professional teaching assignments. 
Other projects have operated on a shoestring budget, in much the manner to which 
university and school personnel are accustomed, scrounging supplies, saving 
money (while wasting time) waiting for buses when a private car would save 
hours, and working on time stolen from regular jobs. The latter procedure 
saves considerable money and nonetheless produces a product— often a very good 
product. What, then, are the advantages and justifications for spending far 
larger sums for more adequate housekeeping arrangements, larger staffs and 
larger work periods? Does the project differential - if any - justify it? 

Do any other outcomes justify it? 



CURRICULUM DECISIONS 

Purpose of the developmental curriculum 

Mission-oriented (e.g. teach disadvantaged to read)/general theme (improve 
math education or better prepare students for college) 

Detailed, behavioral-objectives oriented/more general, nonspecific state- 
ment of purposes 
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Focus - on change in materials used in the schools/change in subject area 
coverage/change in teacher behavior/change in student immediate 
behavior/change in student long-run behavior/change in student values 

Orientation - process/skill/content/values 

What levels of cognitive skills are involved 



Proiects vary in degree of specificity from the very general (e.g. desire to 
"Improve mllLmatios education") to the very specific in te^s of audience 
(e.g. disadvantaged, urban preschool) and skills to be taught (e.g. upgrading 
readiness scores by x points, as measured by standardized tests). This in turn 
delimits the type of personnel, the nature of job assignments and priorities, 
and scope of materials turned out, the funding possibilities, and the freedom 

for exploratory ventures. 



Parameters of the proiect's job ( t he project's definition of its role), and t he 

nature of the materials produced and a target audience 

Definition of curriculum and its ramifications - narrow (the curriculum = the 
textbook)/broad, all-encompassing view of curriculum 



Target audience 

Students- bright/average/slow / • • 

Teachers- well-prepared/average preparation/ minimal preparat o 

Schools- well-equipped/average araduate 

Trainer of teachers- subject matter specialists/Educators, 9 

level/undergraduate level; university/college/junior college 

Decision to prepare materials/no materials prepared 

Type of materials prepared 

Student/teacher/trainer of teachers 

Seqilntial^Unlluding skills taught earlier)/independent units 
learning and teaching materials needed for course 
Degree of curriculum engineering 

Mliriateila{r?ht^ilacher needs, to permit choice/less than teacher 
needs, requiring filling out ^ 

Single medium (e.g. text or f iWmultimedia) , , 

Radically different materials/modest change/combination o 

area at several grade levels/ several subject areas at several 
grade levels 
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Interdisci pi inary approach/multidisciplinary appro ach/unidi sc i pi inary 
approach 

Extent of teaching of tool skills non-specific to the subject area 
(e.g. communication skills) 

Supplement existing materials/replace existing materials 
Grade level - same as before/changed grade level/ungraded 
Individual progress materials/class progress materials 
Materials that are self-sustaining or can be taught by present 

teachers/require some teacher special preparation (e.g. a teacher 
guide) Require substantial teacher retraining 
Subject now taught as discrete discipline/subject not now taught or 
not now taught as a discrete discipline 

Concern only with preparation of materials/concern with implementation of 
materials 

Target audience 

Local/regional/national/international 
College-bound/vocational 
Suburban/urban/ rural 



The decisions about what the project is going to produce reflects various assump- 
tions about the purpose of schools in a democratic society, who should be educated 
and for what, how students learn best, and a relative hierarchy of values.^ For 
example, a decision to produce discrete units rather than sequential materials 
indicated either a lack of concern with transferability of skills or a tacit 
rejection of the notion that -transfer is more effectively taught by giving 
increasingly sophisticated practice in a variety of situations^ ( i.e. spiral 
curriculum). The preparation of prescriptive materials might indicate a lack of 
faith in competency of teachers and/or a belief that fundamentally all students 
learn in the same fashion. The decision to prepare only student materials and 
a subject-oriented teacher guide reflects the assumption that changing the book 
changes student learning and student behavior in the desired direction. Special 
teacher preparation institutes limited to orienting teachers to the new subject 
matter may reflect the assumption that changing subject matter background will 
change teacher behavior in the classroom. Thus, the project's definition of its 
role and the nature of the materials produced reflect some basic though perhaps 
unrecognized beliefs about the purposes of education, how children learn and how 
behavior is changed. This in turn raises questions of internal and external 
validity. 



Basic educational philosophy, psychology of lea rning and assumptions underlyina 
the curriculum and the organization 

Time orientation- the past/present/future 

Concern with present society needs (utilitarian)/future society needs (social 
reconstruction or simply what will the future require) 

Faculty discipline theory of learning/other theories of learning 

Concern with present student needs/concern with adult needs 



ERIC 
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Concern with motivation/unconcerned about building motivation 

Democratically oriented (uniquely suited to schools in a democracy)/ suited 
to any modern society 

View of purpose of schools in the United States- college preparation/job 
preparation/personal adequacy now and in the future/citizenship in 
a democracy/general cultural background/preservation of heritage/re- 
building of democracy 

Views on how children learn most effectively 

Views on how children retain skills and knowledge best 

Concern primarily with the cognitive domain/concern at least equally with 
the affective (attitudes-values) domain 



Some assumptions inherent in the materials and their impl ^mentation include — 

Children are basically different/children are basically the same 

Children learn in different ways/all children can learn in the same way 

Subjects should be learned for their cultural value/subjects should be 
learned as prerequisites for further learning 

Each subject area is responsible only for development of knowledge and 

skills in its area/all areas are interrelated and should be mutually 
supportive and consistent in approach 

Knowledge jger and tool skills are important/higher level cognitive skills 
are important for all students 

Teaching of values is important/teaching of desired values will come auto- 
matically as byproducts of other education. Teaching of desired' 
values can be done in units developed for this purpose/teaching of 
values must be approached in other ways 



By and large, the broader implications of the materials being produced in terms 
of purpose of schools in a democratic, modern society are not considered, and 
the possibilities of functional and dysfunctional outcomes not included in the 
stated intents of the project are overlooked. Thus, developmental curricula 
are generally viewed as relatively discrete entities, sometimes concerned with 
sequence within the subject areas from grade to grade and sometimes not, but 
rarely considered as part of a complex, ongoing social structure in which each 
part regardless of subject focus of necessity influences every other part. 
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Preparation of materials - process and timetable 

Short term deadline on materials preparation/open-ended effort 
'Preparation of experimental trial editions/immediate work on final edition 
Tryouts during writing period/tryouts at end of writing period/no tryouts 
Single writer/coauthors/writing team (large/small) 

Degree of communication among writers and with others during entire writing 
period 

Who prepares outline or story line and when (in advance of hiring writers 
and/or making writing assignments 

Feedback channels during writing period 

Who handles materials last 



While the "writing conference" or team method of preparing materials is one of 
the earmarks of the curriculum project movement, there is wide diversity in the 
method of approach to the "team", with varying outcomes. Some teams are made up 
exclusively of subject matter specialists, some of relatively untrained (in the 
subject area) teachers at the grade level for which materials are being prepared, 
some of combinations of specialists in subject matter and well-trained teachers, 
and- some include science writers. Some "teams" are really independent efforts 
of individuals given discrete writing assignments, with the writings then amal- 
gamated and called a team product. Some teams work in close proximity, with a 
high degree of formal and informal interaction and constant feedback; for others, 
while the writers may see each other constantly, there is virtually no feedback 
on the product until completion. In some teams, there is a status hierarchy, 
with tne subject matter people at the top, the teachers at the bottom, and the 
ranking of the subject people is by professional stature; for others, there are 
no status barriers in the work. 

The question of productivity of the project does not appear to be directly re- 
lated to the number of participants in the writing or to the length of time 
alloted to the writing. On the contrary, for some projects, the lack of a firm 
time limit on the writing (such as a summer writing conference would have) may 
be a barrier to completion of materials. 



Format of materials 

Hard cover/soft cover 
Expendable/reuseable 

Several separate items (e.g. lab manual, text, workbook)/one unit combining 
all materials 
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Format - cards/looseleaf/bound 

Quality of reproduction - commercial quality (printed, color, etc.)/ 
economy type 

Media used - text/lab/programs/computer assisted instruction/games/films/ 
loops/etc. 



Method of release of materials 

Public domain/private domain 
Who holds copyright 
Who gets royalties 

Who distributes materials - project/university press/commercial press 
Free licensing 



The format of materials and the method of release, particularly the question of 
who holds the copyright and the disposition of royalties are ’’sleepers” in a 
sensitive situation. While few recognize the import of these items in terms of 
availability, acceptability and diffusion of new materials, a project whose 
function is the development of new materials can hardly be successful without 
eventual acceptance of its materials in the marketplace? Simply the ambiguitier 
and uncertainties of the copyright question for materials produced with grants 
from Office of Education or National Science Foundation can severely delimit the 
availability of materials to consumers. 



Evaluation of the curriculum 

Systematic, formal evaluation/informal, nonstructured evaluation 

Short-term evaluation/long-term evaluation (longitudinal and follow-up 
studies) 

Formative evaluation/summative evaluation 
Microevaluation /macroevaluation 
Evaluation of product/evaluation of process 
Internal self-evaluation/external evaluation by others 



All projects evaluate in some fashion, but often the evaluation is informal, ^ un- 
systematic and perhaps even unrecognized as evaluation. There is a wide variation 
in the degree to which evaluation is formalized, in the evaluative techniques 
used and in the ways in which these are used. Radically new curriculum evaluation 
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techniques are emerging slowly as there is recognition that many of the 
traditional approaches to educational research are inappropriate for develop- 
mental curricula. However, in many instances the new approaches are still 
relatively crude and even where data are carefully and imaginatively collected, 
their' use is generally non-systematic and far from optimal. Thus, while projects 
are learning to collect feedback of various sorts for purposes of improving their 
experimental projects - a major departure from traditional methods of revising 
materials - they have not yet learned how to use it systematically. Furthermore, 
there has been far more interest in - and more adequate funding for - formative 
evaluation than summative evaluation. 

One of the most difficult of the questions concerning evaluation of developmental 
curricula concerns selection of appropriate standards to use for evaluative 
purposes, and whether the target audience accepts these standards. If a product 
is radically different from earlier products, to what extent should prior standards 
be used for judgment? This is not only a matter of use of standardized tests 
and other conventional measures of student achievement but extends to such questions 
as judging teacher performance in the Individually Prescribed Instruction (IPI) 
program or in an independent study situation and the relevance of achievement 
measures to attitude objectives and of short-term measures to long-term global 
objectives. 

One of the major innovations of the developmental curriculum movement is the 
preparation of experimental materials which have tryouts before materials are 
issued for general use. The method of tryout varies in terms of* 

with whom (nature of students) 

by whom (nature of teachers) 

for what purpose 

how long 

how many times 

when (at what stage of materials tryout) 

feedback from whom (staff observers/teachers/students/etc. ) 
degree of change possible at this point (i.e. to what extent 
are materials jelled) 
degree of control over tryout conditions 

Some projects use "instant children", children available on short notice to try 
out small segments of materials for feasibility. Some wait till preliminary 
materials are completed and then try them out during the regular school year, 
either with neerby schools or with schools in a broader geographic area. Some 
projects depend primarily on staff observation for feedback; others depend more 
on teacher reports of tryouts. Some have staff members do the teaching for the 
earliest tryouts; others use regular teachers with special orientation preparation; 
and still others use regular teachers who have no special orientation beyond 
some written teacher material. Some tryouts are genuinely concerned with feed- 
back; others are window dressing, so the potential purchaser can be told that it 
has been successfully taught in situations similar to his. 



o 
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Traininq for use of materials 

None needed/self-training materials included/special training required or 
advisable 

While training is required or advisable- 
Who does training - project/other 

Who selects trainers - project/other (e.g. founder); what are criteria 
for selection 

Background of trainers- subject speclallst/Educator 

Who selects trainees - project/other (e.g. founder, university); what 
are criteria for selection 
Length of training - short course/extended period 
Timing of training - before teaching/during school year/both before 
school year and during school year/both before and after school 
year 

Orientation of training - subject matter orientation/methods orientation/ 
combined subject matter - methods orientation 



Implementation of materials following completion 

Some projects have taken as their mission the preparation of improved mater- 
ials. (This may reflect the tacit assumption that existence of good materials 
will result in their use and in the change of teaching and learning.) Other 
projects, either initially or after some experience with materials preparation, 
have taken a broader concern with materials implementation, including a concern 
with widespread distribution, appropriate inservice preparation, change in pre- 
service teacher preparation, appropriate use in the schools, extent to which 
materials are adapted, continuing rather than short-term use, conditions conducive 
to curriculum change, problems of implementation, etc. 



o 
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Some Decision Points and 
Alternatives In Developmental Curricula 



lt«octi*n 



To Tho Grobmon Popor 

by George Maccia 



One can only applaude Professor Grobman's concern that decisions direct- 
ing the course of developmental projects in education be reasonable. ^What 
will follow her first steps toward a sociology of knowledge of educational 
development programs will be of interest to all of us. Clearly the^delinea- 
tion of variables is a requirement for an adequate theory of educational de- 
velopment, but such delineation in itself is not enough. As I am sure Pro- 
fessor Grobman is aware, an adequate characterization of the ^relationship 
between variables is most essential. Her listings at this time suggest the 
complexity of such a characterization. I trust she and others will _under- 
take the task of developing adequate knowledge of developmental projects in 
education . 



Ditcuiiion 



Dr. Grobman made the following comments about her paper. 

1. A developmental project creates an emergent curriculum through a 
systematic process. There is too little concern among curriculum developers 
(and others) about the aims of education, that is, for making the value 
judgments reflected in the objectives of developmental efforts. Uncoordinat- 
ed projects, each dealing with highly specific outcomes relevant to particular 
disciplines, are very likely not to "add up" to anything very significant, -or 
add up in a unified way. The impact of a single, discipline oriented project 
is apt to be very restricted. Perhaps we need to change whole school programs 

if a large overall impact is to be made. 

2. There is too little attention paid to considering the negative out- 
comes of learning. 

3. A considerable part of the discussion which followed centered pri- 
marily'upon two related questions; a) Who makes the value (or^large policy) 
decisions concerning educational goals and how are these decisions made? 

(It was pointed out that we do not now possess any adequate decision procedure 
for deciding between values.) b) How broad should the scope of developmental 
projects be in order to reform schools in desired directions? 
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Def inition 



A developmental project in education is an active, visible effort to 
carry out a plan which promises to remove a deficiency in some aspect of 
learnrer performance. An essential attribute of the definition as opposed, 
say, to that of a research project is the emphasis upon change in a learner 
or population of learners. A research project can be characterized as an 
attempt to close a gap with respect to knowledge in a discipline. 

At times, persons in schools engage in a chain of projects which are 
only remotely connected, if at all, with performance of learners, e.g., 
campaigns to pass bond elections for funds with which ^ to finance higher 
salaries in order to attract teachers with qualities necessary to 
produce desired changes in learners. Many "enabling" projects aimed at 
intermediate goals don't make much difference in the ultimate performance of 
learners even though they appear to have a logical connection.! When the 
link between project and learner cannot be empirically established, the project 
lacks educational validity. For instance, projects undertaken in response to 
demands of teachers which can best be justified in terms of teacher morale 
rather than pupil welfare are not valid educational projects. Validity would 
exist, however, if it could be demonstrated that pupil progress is improved 
as a result of an intervention that also raises the morale of teachers. 



Project Planning 

Planning of developmental projects occurs at three levels of remoteness 
from learners;^ (a) societal (national, state, and local governments)} 

(b) institutional (school); (c) instructional (classroom). At the societal 
level a project is in response to a Federal interest, e.g., a job corps 
project to equip workers with given salable skills and curriculum projects 
to provide learners with the spirit and content of the disciplines. Usually 
a National project does not specify the individual learners to be changed but 
rather names the class of learners eligible for the treatment. Projects 
drawn at the institutional level have as their target population those wAio 
hold membership in the school as learners. A project to change the behavior 
of parents or teachers, for instance, would not qualify as developmental 
in education unless it was undertaken to produce corresponding consequences 
for pupils. - When a classroom teacher plans and conducts a developmental 
project, it is aimed at particular individuals, as in the case of a teacher 
who devises and tries out a system for differentially rewarding John, Mary, 
and Jose in order to maximize their participation and achievement. 



1. Gideonse, Hendrik D. "The Relative Impact of Instructional Variables, 
The Policy Implications of Research," Teachers College Recorj_ , April, 
1968, Vol. 69, No. 7, pp. 625-640. 

2. Goodlad, John I., The Development of a Conceptual System fo r Dealing 
with Problems of Curriculum and Instruction , Cooperative Research 
Project 454, U.S. Department of Health, Education, and Welfare, 1967. 
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Plannina the Pro.iect 



Problem Identification 

The first step in planning an educational project requires that one have 
a clear expectation as to what learners should be able to do, i*e«, a concept 
of desirable and desired performance in a situation or class of situations-- 
social, intellectual, physical, vocational. Next, data (observations) are 
collected in order to note the actual performance of learners in the specified 
situation. The gap between desired and actual performance becomes the basis 
for formulation of the problem. The decision whether to close or partially 
close the gap is a difficult one. How much of an improvement in the learner 
will be satisfactory? How many learners must change? Is it enough to show 
that a statistical diff erence will occur in learner scores following a^^ 
project? Or must one state in advance what constitutes a practical difference 
by which the findings of the project can be judged as worthwhile? How much 
of a difference must the project make in the behavior of the learner in 
relation to the cost of the product before one would recommend that it be used? 
On one hand, we have the obligation to look at the finished product, e.g., 
filmstrip, set of procedures, a system of instruction, coming from a develop- 
mental project and to ask ourselves* "Does the change that will occur in 
learners from the use of this product warrant the multiple costs of reproduc- 
ing, introducing, and using the product with learners?" The planner should 
consider initially whether the contemplated product, even if it meets his 
specifications for learner change, would actually be feasible in practice, 
e.q., if an instructional solution to a school problem is likely to be 
demonstrated but the anticipated difficulties of acting upon the solution would 
prevent its adoption, then the project should not be undertaken in the first 
place. On the other hand, we must ask whether the developmental costs ^mosu 
likely to be very much higher than the reproducible costs of a finished 
oroduct) are warranted even if the product turns out to be of no value 
(learners don't change) and the best that can be said is that the enterprise 
showed what wouldn't work. Few project developers are able to P^eaict in 
advance what the results will be. Developers lack normative expectations 
when dealing with tasks that are not repetitive and with learners who are 
not uniform. Further, to the extent that new methods and materials are used 
in solving the problem, one does not know beforehand the power of his innova- 
tion. The designer can, however, specify in advance what ^®sults 
present as evidence that the project has or has not been ;vorth the time and 

expense. 



Plans for Evaluation 

Should plans for evaluation precede plans for solving the problem? We 
have said that one should state in advance what will constitute evidence 
that the objective has been reached, i.e., the problem solved. Objectives 
are clarified when the actual measures to be used are specified: t®®*, 

semantic differential, guide to observations, interview ®®hedule, and rating 
scale. It is especially helpful to stipulate the level of acceptable 
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performance by the learner on the instruments chosen. By stating the criterion 
in advance, one should be better able to analyze (identify) prerequisites 
required for attainment; thereby increasing the likelihood that the proposed 
section includes necessary components. The selection of learning opport- 
unities (means) logically should follow the clear specification of the 
Tsiied ends" JUS? as a teacher should offer pupils opportunity to practice 
behavior relevant to both prerequisite learnings and that demanded by ^ 
final test (provided the test represents the teacher s 

activities of a project be relevant to expected outcomes. Analysis of the 
criterion gives a clue as to what must be present if the solution is to have 

its intended effects. 

Sometimes directors of projects do not act ^^befo^e® 

recommendation. Instead, the activities and procedures ®^® 
the criterion measures are determined. If one wants to take little 1 k 

of failing, this might be a good way to proceed. High P®^f°™®"®® "“t®rial, 
can be obtained by indexing the learning opportunities--activities, mat , 

assignments, etc., with which learners are to ®"f 9® ^ 

then selecting these same opportunities (or samples f t*'®. 

of opportunities) to form the post-test. The more c °®® T 4 . 1 ..,,. ig gf show- 

measures resemble the training program, the greater chance there f h 

ing that the program makes a difference (is effective). A ®^“erion test^^^^__ 

composed of items equivalent to those met in training ts 

for finding whatever differences there are because of training. 

Doint ocoured recently when a school developed a project to improve t 
??aSLrp?o?iciencrof children. The proposed solution included a nj»ber 

Sion skillsi^but all iteL were identical to those items children would practice 
di??ng ?ie instLctional phase of the project. Prior to ®®truction 
each child received a randomly drawn sample from the 400 items. Mean scores 
for groups of children were compiled and instruction commenced. After the 

Considerably irCrcCCs^CrrCf cCntCCl gCCup^whC received instruction under 
conv©ntional arrang©m©nts. 

The foregoing brief description of ^itera-samp^^ as 

about close agreement between cti y ^^ the 'technique of item-sampling 

“2,- P.n d»nd ».t ■ WV 

a principle lo new i®®tances , both during traini^ CUCeM oCC iT^teTlTlli 
(b) Should success of the l„^Clwd il the planning enterprise? 

sCCirL'f dIluCCuL°L?wCen the roles of formative and summative evaluation 
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is helpful here.^ Data collected to reveal the effects of early versions and 
initial steps as well as data indicating the power of the terminal product 
is necessary to development (the guiding of revisions and new formulations), 
hence the term formative. Such data enables the project staff to modify the 
proposed solution so that the intentions are fulfilled. Feedback in terms ^ of 
learner responses to the stimuli presented enroute and on the finax criterion 
measure indicated the degree to which overall results desired by the planners 
are attained. The role of summative evaluation is to help one (usually a 
consumer) decide whether the results satisfy a broader range of desirable 
consequences than those necessarily sought by the planners.^ Often a non- 
planner is concerned with specifications which were not of import to the 
designer, e.g., presence or absence of certain aesthetic and political factors, 
and comparison of the product on the basis of efficiency and cost effectiveness 
with products from other projects purporting to fulfill a similar purpose. 



Proposed Solution to the Problem 



Review of the Literature 

What literature should one review? — writings pertaining to the nature of 
the problem itself and sources treating strategies and research^ tools for 
solving similar problems. There are several reasons for reviewing the 
literature before designing a solution. It may be economical of time to learn 
how others have met similar problems and with what success. If, for instance, 
there is a programmed filmstrip already which purports to close a gap of concern 
to the planner, it should be checked out. There is little value in 
what already exists. Development undertakings in schools are characterized by 
an absence of cooperative augmenting and revising of existing programs. 

Extension of the work of others seldom occurs. 

Review of the literature also helps one better understand the Problem. 

That is, new dimensions and definitions of the topic may be 

that will help the developer make a better task analysis, i.e., identificat 

of prerequisite behaviors necessary for the learner's successful perform n . 

Further, review of instructional variables and selected characteristics o 

learners is a useful supplement to a review of the literature regarding the task 

itself. Projects which attempt to change learners indirectly through 

administrative decisions may find these factors worthy 

parental cooperation, procedures for teacher se^ecuion an« apprais 1. 

which are to be directly applied to learners in the classroom may make 

use of such variables as the following: number and kinds ox examples, 

competitive versus visual stimuli, oral versus constructed „ 

immediate versus deferred knowledge of results, prompting versus confirmation. 



3 . 



Scriven, Michael, "The Methodology of Evaluation," Perspectives o |. 
Curriculum Evaluation . AERA Monograph Series in Curriculum and Evaluation, 
Rand McNally and Company, Chicago, pp. 39 - 83 , 1967 . 
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Instructional variables have a better chance of showing their power if 
they are paired with learners of given characteristics; (a) learners with a 
predisposition for a given style of learning— through ^reason, space relations, 
or verbal, (b^ learners who have been rewarded or punished in the past tor 
engaging in particular activities such as working alone or with peers (knowledge 
regarding values held by the socio-economic groups from which the learners co 
may be helpful here), (c) learners who are exceptional in their chronological, 
physical, or mental state— blind, deaf, aged, anxious ^f^^us non-anxious. 

We are not asking that the planner consider learner, task, and 
variables as important in themselves. They must be related if planners 
to generate a more effective solution to the problem. Treatment, e.g., 
instructional procedures and materials, must be linked theoretically and 
operationally to both task and those characteristics chosen as crucial 
describing the target population. 

Need for making clear why the solution shoul d produce intended consequen c e . 

This guideline might be unnecessary were it not for the fact ^hat in 
educational practice projects often fail to show a logical ^oeoretical) conn- 
ection betweL the proposed solutions and the intended outcomes. ^ suspect 
that often teachers and administrators have personal motives 
projects which are sometimes more important than their manifest t 

or publicly acknowledged) purposes. Faculty have been kno^ to resurrect 
their old and favorite plans and to offer them as answers to current Y 
popular problems. Some persons may be motivated by the P^^^pect of getting 

funds to carry out their own pet ideas even if it ®®7be 

to the problem for which funding is provided. Support for ;^his opinion can be 

found in the final reports from government sponsored projects which were 

funStd ?or reso^vLg educational problems but which ended ^th mere sociological 

and psychological information. Also the fact that a ®=hool_system spends a 

fortune on replacing a set of textbooks vdien there is no evidence to suggest 

that the new books ^11 make any difference in either the academic progress 

of pupils or in attitude toward learning indicates that ® 

are not always rational, i.e., that the means are consistent with stated go 1 • 

MPPd for Station How One Will Know that th e Desired Results were brought ab out 
by the Plan and not bv some Concomi tant Factor 

We can say that one has knowledge in instruction when he can produce 
at win desired effLts in learners (he "knows"). Although Production of 
sScrknowledgrif important to the advance of instructional technology (discipline 

hn^to the oroiect developer the solution of a problem is more important than 
the scientific^findings. The use of research procedures to enhance gains by 
learners, however, is^not inconsistent with our position. “ 

bench maiks of learner competency before treatment. Also we must have measures 
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fox guiding dovolopniGnt and fox validating th© pxoduct and th© pxoject# The 
use of non-instructed contxol gxoups, base-line data^ xandom assignrient to 
txeatmentj and othex such accoutxements associated with xeseaxch should 
contxibute to the quality of developmental studies# Popham's Simplified Designs 
fox School Reseaxch^ is an excellent self-instxuctional pxogxam to help the 
plannex impxove his effoxts in pxogxam development# By using the designs in 
this booklet, the plannex will know to what extent the xesults axe due to the 
pxoject and the contxibution of diffexent components in the pxogxam to overall 

effects# 



One xecent advance in educational xeseaxch of special xelevance to pxoject 
development should be mentioned hexe# This is the pxactice of administexing 
two ox more treatments within the same classroom, as opposed to offering each 
of these treatments in a different classroom# By administexing, for example, two 
forms of instructional material, randomly assigned to pupils within a classroom, 
the developer can use the number of pupils in the class as the proper unit for 
statistical analysis in determining the value of each form# This is not allowable 
when the developer has presented one form in one class and the second form in a 
second class# In the between classroom situation, the unit of analysis must be 
the number of classrooms, not pupils# The chance of finding significant 
difference when the N is 2 as opposed to 20 is obviously unlikely# It is 
difficult to equate conditions in classrooms, whereas when two treatments 
axe presented within the same classroom at the same time, extraneous var a es 
are more fully controlled# Developmental studies of an experimental nature 
carried out within single classrooms should show more positive results from 
the use of promising variables than where the effects of the treatment are 
washed out by failure to control for other variables# Once it is shown under 
controlled conditions that the variable (or its combination with other factors) 
is important, the project developer can wonder why the same variable does not 
seem to be vital in less controlled situations, that is, the findings are not 
replicated# This in turn will lead to the discovery of new variables of 
importance in the modified situation# Incidentally, because the reward of 
finding significant differences will occur more frequently when planners use 
within classroom designs, there should be an increase in the number o 
mental projects conducted at instructional levels# The likelihood of finding 
NSD under the prevailing practice of comparing treatments in different 
rooms has discouraged experimental projects at the classroom level# 

Improvement can occur by (a) selecting better objectives, Resigning 
procedures which maximize desired results, and (c) modifying procedures so that 
the same results can be obtained in a more economical fashion. By way ot 
example, the conduct of developmental projects at Mexico's 

Productivity often includes an attempt to produce *he same effect in learner 

but doing it in less time and with fewer resources. This 

the modern technologist’s goal of trying to produce more with less 

and fewer human resources. The attainment of this goal require far+ivl+les 

design. Versions of programs with and without particular components (activities, 

material, and their ordering) must be experimentally manipulated and results 

compared# 



4. Popham, W. James,’ Simplified D esigns for School Resea^h, Southwest 
Regional Laboratory for Educational Research and Development, 1967. 
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One further argument for inclusion of a research strategy in each 
developmental project can be linked to a present trend. Currently, Federal 
priorities in educational developmental projects seem to be shifting from 
the development of curriculum to the development of teachers and other 
personnel associated with instruction, e.g., see Education Professions 
Development Act of 1967. Research might reveal the limitations of this decision. 
The underlying assumption seems to be that learners will show greater gains if 
the preparation of the teacher gets more attention. The question as to when a 
school system should invest more in training its teachers and when to expect 
a bigger payoff by revising instructional materials can be empirically answered. 
However, perhaps a more effective policy would be to focus efforts on assess- 
ment of teachers rather than either the preparation of teachers or the 
development of curricular material. Developmental projects are needed to 
show the value of making sure that teachers are held accountable for the 
progress of their pupils. Stress of accountability might be an intervening 
variable provoking practices such as the teacher's willingness to use both 
new materials and effective teaching procedures which the teacher already has 
but, under conventional conditions, does not regularly use. In other words, a 
developmental study with a research design may reveal that progress in pupil 
achievement is not due to the teacher's lack of knowledge of teaching, nor 
in the school's failure to provide curriculum materials. It may be that there 
exists little incentive for the teacher to use what he already knows. 



Kinds of Developmental Projects i Implications for their Conduct 

Developmental projects differ in their specificity of detail, visibility, 
and the degree to wrtiich they evolve new possibilities. Some projects, such 
as those concerned with the development of instructional products (programmed 
lessons), place much emphasis upon specifying detail and attaining reproducible 
instructional sequences. Other projects put first emphasis upon serving as 
demonstrations--working models to be emulated and adapted. Newer programs 
for teacher preparation are cases in point. Guidelines for reviewing Title 
III proposals tend to classify developmental projects as: 

(a) Inquiry — to formulate new ideas and theories in order 
to solve education problems. 

(b) Invention -- to design, field test, and refine a new 
instructional program founded on research and theory. 

(c) Demonstration -- to demonstrate a model program resulting 
from the invention process. 

(d) Adaptation — to adapt a model program to local conditions 
and needs. 

Projects which have as one of their central purposes the generating of 
aiternative possibilities invite special interest. They are usually character- 
ized by an effort to put together personnel, institutions, and instructional 
material that have been independent from each other. Planners who try to^ 
create new organizational forms (structures) do so partly to upset tradition, 
partly to generate neW" problems, but chiefly to evolve new possibilities. 
Goodlad's rationale for advancement of education as expressed in his presidential 
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address at the February 1968 meeting of the American Educational Research 
Association exemplifies the point of view which would foster developmental 
projects for the evolvement of possibilities. This view appears to be contrary 
to the position taken in the present paper. Those stressing evolvement do 
not demand that the effect of an intervention (invention or 
stipulated prior to its creation. Although they do not specify the effect 
that is to follow, I have a hunch that those who plan for evolvement are dis- 
satisfied if newer possibilities of merit do not emerge from the projfct 
undertaken. However, it is not clear to me how accountability is built into 
such projects. Neither do I have evidence with which to judge the validity of 
such operations for learners, i.e., that such projects really advance their 

welfare. 



In contrast to the advocates of open-ended projects, the product developer 
has been quite precise in spelling out the need for objectives in advance 
and equally specific in providing guidelines for the conduct of such projects. 
Remember that the product developer is interested in a product 
producible in two senses: reproducibility of effect in a target .. 

of learners and reproducibility in the production of the product, e.g., recording 
film, book, set of directions. Examples of guidelines given to product 
developers are these: 

(a) Specify instructional outcomes sought 



(b) Define prerequisite tasks 

(c) Be concerned with learners' affect toward the program 
as well as with achievement 



(d) Try out each version with a few learners 

(e) Note results and revise the program when it fails to 
elicit results set in advance 

(f) Use a variety of approaches in presenting stimuli and 
eliciting responses, even go beyond what others have 
done to get results 

(g) Consider the context in which the product will be used 
Make the product an acceptable part of the larger 
system of instruction 

(h) Prepare programs only when the objective is important 
for large numbers of learners 



Examples of Problems in Project Development 

■ Two considerations are singled outt one, those pertain^g f 
of tryout and, two, those related to operational analysis. The subject o 
tryout raises issues and problems such as the following: 
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(a) What constitutes an adequate sample of learners? 

What are the relative values of using small or 
large numbers of learners in the pilot study? 

(b) How can one make sure that the situation in which 
the tryouts occur is generalizable to the ultimate 
population of situations where the product will be 
used? 

(c) How can the developer know for sure what he is really 
trying out? For example, has he controlled for 
contaminating influences? (e«g«, enthusiasm of an 
assistant which may be provoking important effects.) 

(d) Do data gathered from sample lessons (prototypes— 
sampling part of an envisioned program) allow one to 
really predict what the ultimate total program will 
be able to do? 

Operations analysis occurs in a larger frame of reference than that which 
focuses on the development of the particular project. Its purpose is the 
improvement of procedures for project development. In some instances, these 
procedures may be generalizable only to the local activity; in other instances, 
the insight won will be most helpful in subsequent projects of the same kind 
wherever they are carried on. Operational analysis requires that a record be 
made during the project by which one can reconstruct the step that seemed 
to block progress and identify those procedures which enabled the project to 
move, forward. Most of us have observed informal efforts to assess the strengths 
and weaknesses of one's way of working. Usua-’-ly these "self assessments" 
are spotty. How much change in developmental processes might be expected 
under two conditions; 

(a) Those where systematic records (observations) are made 
of ways of working with accompanying indications of 
delay and advance in the project; 

(b) Those where there are no provisions for formal analysis of what 
happens and with what consequences during the conduct of the 
project? 



Summary and Conclusions 

The comments in this paper reflect a point of view regarding developmental 
projects in education. This view gives emphasis to the need for designing 
and assessing projects in terms of changes to be made in learners. It is ^ 
argued that the practice for judging a project on the basis of external criteria 
(changes in learners) as opposed to reliance upon some internal quality present 
in the project (procedures of quality and style) should lead to educational 
betterment. While advocating the desirability of stating intended outcomes 
in advance of project planning, we believe that an alternative strategy-- 
that of withholding stipulation of effect until interventions are created— 
should be examined. 
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Specific problems in the conduct of developmental projects have been 
identified along with descriptions of practices for meeting these problems. 
Guidelines have been given which seem to hold promise for generating solutions 
to educational problems through developmental projects. It has been advocated 
that a research strategy appear in each developmental project. 

The question now before us is: Can we use the issues raised in this paper 

together with guidelines proposed and the description of problems and practices 
to help shape a model of procedure for planning developmental projects? 
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Discuffion 

Dr. McNeil presented his paper by touching upon certain selected content. 
The subsequent discussion included the following comments and questions. 

1. It was suggested that we need a topology of objectl’-'es for the learner 
and/or the educational system. The developmental process should not be guided 
exclusively by pre-determlned, desired learner outcomes. 

2. We also need to take into account the foreseeable, indirect and concom- 
itant effects of the program on the educational system, as well as upon the 
student. Knowledge of all of the outcomes and their consequences should affect 
development. 

3. Evaluative procedures should be capable of attributing program out- 
comes to program characteristics (accounting for other relevant variables) . 

4. An early consideration in planning a developmental project should be 
the attempt to make the product acceptable to the education system in which it 
hopes to function. 

5. Research and development projects usually arise from one or more of 

the following incongruities: (a) Educational Inputs and desired outputs; (b) ^ 

existing knowledge and existing practice; (c) social values and social realiti- 
es; (d) relative position today and creating the future we would like. 

6. Questions related to whether developmental projects should be central- 
ized or decentralized included: (a) How should resources available for a pro- 

ject be distributed? (b) How can quality control be applied? (c) Are adequate 

numbers of qualified staff available? 

7. Quality control during the developmental process asks three questions: 
(a) Does curriculum content have authenticity? (b) Does the curriculum (or 
product) achieve its expected outcomes for the target population? (c) Is the 
project being conducted efficiently? The first two questions are answered by 
formative evaluation, which should be conducted, at least in part, by persons 
who work closely with the "writers" or project developers. Theremay be a 
conflict among the measures of quality control, therefore, their relative 
emphasis should be decided early in the project. 

Dr. Maxwell opened the evening session by introducing the questions which 
had been suggested as guidelines for the conference. Question 1 (What is an 
educational development project?) was discussed with the following points, 
among others, being made. 

1. A developmental project is a group of modified cognitive structures 
and related instrumentalities designed to increase the effectiveness and- ef- 
ficiency of a given program. The resultant program should have a utility 
base outside of the cognitive domain of education. 

2. Developmental projects produce materials, techniques, processes, hard- 
ware, and organizational formats for instruction which accomplish certain ob- 
jectives, specified in advance, and which are construed to be part of the 
broader goals of education. The products are immediately applicable with stu- 
dents, yield reasonably predictable results, and contain a certain degree of 
re-structuring of applied knowledge. The process must also include evaluation. 

3. Development engineers knowledge and theory into immediately usable 
educational systems. It includes evaluation of the product in terms of the 
achievement of educational goals, and it leads to further- research and devel- 
opment by identifying gaps in knowledge and practice. 
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4. The research process attempts to discover or verify knowledge and to 
develop or test theory. It concludes with something known. 

5. The development process contains a large research component, e.g., 
evaluation. 

6. Development translates research results into usable instructional 
forms, and'piovides data on the performance of these forms, such that those 
who decide upon educational policy and practice can make more rational de- 
cisions . 

The evening session concluded with no one appearing to be completely 
satisfied with all of the stated definitions. There was, however, a feeling 
that the group was expressing the essence of the concept of development, and 
that it agreed upon that essence. 
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For several months I have followed the pages of Science attending to the con- 
tinuing discussions of research and science policy-* I have generally concerned 
myself with the degree to which the policy questions which arise in educational 
research and development can, must, or ought to be considered as a subset of 
those which arise for science as a whole. Thus it has been with no small degree 
of interest that I have read the articles prepared by Philip Handler^, George 
Daniels^, Harvey Brooks^, Lee DuBridge^, and Michael Reagan^. 

Coincidental to this continuing dialogue, I have been for the past three years 
engaged in a general effort to improve the quality, effectiveness, and impact of 
the research effort in education. No small part of that time has been spent 
grappling in practice with many of the same kinds of issues as receive explica- 
tion in the articles cited above. For some months I have been thinking about 
research and development in education trying to develop a model which would ex- 
press the different functions within the total research effort, the various 
sources of initiative for these different kinds of activity, and the relationships 
among both the functions and the sources of initiative. My thinking has, of 
course, been in no small measure stimulated by recent debate about the ways in 
which schools, instruction, and education are likely to be improved most quickly 
and with the most substantial cumulative and lasting impact. 

Reviewing the several statements cited above in terms of their general useful- 
ness for spotlighting and clarifying analogous problems in the field of educa- 
tional research revealed the particular value of Professor Reagan's analysis as 
a summary of points of view already advanced and as an identification of the 
generally individualistic starting point of many of the theoreticians of science 
and research policy. The model presented in this article constitutes, I think, 
an instance of exactly that for which Prof. Reagan called , namely, a discussion 
of research and development as seen from the point of view of someone who sees 
himself as a sponsor, planner, or user of research and development. It was not 
developed directly in response to Reagan's suggestion; in fact, it was only after 
a colleague recalled his article with specific reference to the similarity of view 
on the research and development distinction that I realized how closely what 1 
had formulated came to responding to Reagan's suggestion for some dialog on the 
part of users of research to more adequately balance that which had been gener- 
ated by the performers. 
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One Logical Modsl of ths Chan ge Process 

Typically, the context for discussion about research and development for 
education is a description of the change process in education, \arious models 
of change have been proposed. The one which seems to have the highest degree 
of currency at the moment places research and development in a linear arrange- 
ment beginning on the left with inquiry (research) and proceeding to the right 
through development, diffusion, and adoption (see Figure !}• 

In proposing their model, Egon Cuba and David Clark called three caveats 
to attention. First, they noted that the model was constructed on logical 
grounds and that it was largely unsupported by empirical research. Second, they 
pointed out that it was not necessary for change to begin at the research o 
inquiry stage. Third, they noted that the model itself was a uni-dimensional - 
analysis of change roles which are influenced by a multi-dimensional range of 
variables not entirely accommodated by the model's structure. 

As a model of the change process, this particular schema has the virtues of 
being simple and logical. However, those of us who have worked 

problems of research policy in education see shortcomings. The ^ firLtions 
does not emphasize sufficiently within its structure that initiative for actions 
ff"di??L:rkinds can take place at any point in their 
initiatives may come from locations other than where tne act 
b^peSed! Because of its linear nature, and despite the second caveat 
mentioned above, the model unwittingly implies that innovations begin with the 
findings generated by fundamental research. 



The Output Model 

The purpose of developing an alternative model is to create an heuristic 

which (a) illustrates the essential differences^bet^veen^rese^ 

activities and (b) shows how the two are • , , rmnh+ to illustrate different 

*:u*^:s°rinu?at":iti^^^^^^^^ f^begi:nfn^ ea^ious activities. It should 
be able to show or imply the interplay among all the functions in the e 
improve instruction and education. 

The outDut model is based on the conviction that research, development, and 

school operations can be viewed as^distinct^k^ 

elch kinHl'Ictivity a?e the results of decisions based on different kinds of 
data and equany distinct kinds of internal and external needs. The mode implies 

that while there may be a strong logical flow Settings, 

devflo^ent programs, which, in turn, reveal the need for certain basic inform- 
ation and theory. 
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Figure II depicts the model. Three planes are shown, each symbolizing 
the different orientation of activities conducted under research, development, 
and school operations. (The model is - as are all models - an abstraction from 
reality. In the real world, these activities are not always neatly separated 
either in time or location. The point of conceptually - and Therefore 
graphically - separating them here is to illustrate the essentially different 
orientations of the three types of activity and the consequences of those diff- 
erences.) For each activity represented in Figure II, the model depicts an 
initiative leading to an output characteristic of that activity. 

The lower plane symbolizes the knowledge orientation of research; the object 
of research, of course, is to generate new knowledge. And, as is also well 
understood, one of the significant features of research is that when an activity 
is begun the specific outcome is not known . For research, C represents an 
initiative undertaken which culminates in a research finding represented by Fc. 

The middle plane symbolizes what I call the process orientation of develop- 
ment. The object of development is to produce materials, techniques, processes, 
hardware, and organizational formats for instruction which accomplish certain 
objectives, specified in advance, which are construed to be part of the broader 
goals of instruction or education. On this view, one of the significai fc features 
of development distinguishing it from research is that when an activity is begun, 
the objective is known or established at the outset . The objectives for a 
development project, ideally, are cast in the form of performance specifications 
(PS), and all activities are geared to producing the necessary products and 
processes which will meet those specifications. In Figure II, B indicates an 
initiative undertaken for development culminating in the creation of a process 
which meets performance specifications PS^. 

The top plane symbolizes the activities characteristic of school operations. 
The operating educational system can be said to be production oriented. Thus, 
the object of school operation is to act upon human beings in order to train and 
develop in them various skills, attitudes, beliefs, and knowledge calculated to 
serve both society and themselves. Certainly one of the significant features of 
initiatives in school operations is the weight of the responsibility on the 
school administrator for choosing the right kinds o_f processes to achieve the 
outputs that society and individuals specify. In Figure II, A represents an 
initiative to install a process leading to the production of education output EOg. 



A "Walk-Through” of the Model 

To illustrate the relationships among the three types of activities, consider 
the following example. A responsible school official, faced with evidence that 
certain outputs desired by the society are not being achieved for a significant 
portion of the children in his charge, searches other school operations and on- 
going or completed development projects for processes designed to meet his need. 
Should he find nothing to suit his particular problem (e.g., the low reading 
achievement of culturally disadvantaged children), he may then exercise his pre- 
rogative to call for the initiation of a project to design and develop a process 
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whose performance specifications are such that upon installation of the process 
in his school, it will yield the desired educational output (e.g., increased 
level of reading achievement in the target population). 

Once the initiative for the development project has been undertaken and the 
performance specifications established, the development project then ® 

search for relevant research findings which may offer clues to guidethe develop- 
ment project. (Whether or not this step is taken after the project is begun or 
immediately before is not really important. What is crucial tha a 
point near the very beginning of the effort such a search is made). 
with a particular finding (perhaps, for example, the great 

attitude on student achievement as revealed by the survey conducted as P^it of 
+V.O Pinwden studv®) the project might decide to develop a process which 
eratfir^ries to enge^^ measure of parental involvement in home instruct- 

iona! Lperiences which, in turn, are carefully geared to complementary 
l^the school setting. Having made that decision, the developers might then 
discpver^hai they require further inform.ation about the 
parent-child interactions to stimulate maximum learner f 

fur^herdatf io guiL t^^ of materials. When useful findings are 

identified they^can be incorporated in the development effort which then proceeds 
if a succe^sfu! conclusion. When, using iterative techniques of design, develop- 
mpnt trial and redesign on the basis of feedback, materials encompassing ^ot. 

exf riences’and parft-child interactions in the home are ^ successfully developed 
and validated, they may then be transferred^ to ^the operating netting ^where h 
administrator may install them as part of his instructional progr 

This example is illustrated in terms of the model by Figure III. #1 

childiL). This d^manf creates pressure on the school administrator to respond 

wiih some sort of initiative. That initiative s f df^‘ no“ 

symbolizes his search for an effective process to install. Since he did 

fi^Id it, his response was to call for a at #4)*s!ch that 

next step was to develop the performance specifications (PSx ^ J soecif- 

?hey corLspond to the educational output desired by society Once the specif 
ications for the development project are established, the next ®tep is to sur y^ 
r:!ated research^ seeking guidance for the f 

ducted and the finding (Frx) re sianificance of parental attitude) is 

the desired educational output (e.g.,_ 9 additional research 

incorporated into the incorporation Jf relevant findings 

assistance is symbolized by D at ffb. ine inco p effort is symbolized by the 

(again Frx) from that initiative into the Z completiL of 

solid arrow to the development line ^ of the process into school operations, 

and #!rirprodSc?ion*of ?he des^red'output, higher levels of reading achieve- 
ment, as a consequence. 






FIGURE III. A "WALK-THROUGH" OF THE MODEL 
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A Sampling of Other Possible Interactions 

The depiction of a sampling of other possible interactions among research, 
development, and operations can be found in Figure IV. 

Example: A school official feels the need to assess the degree to which insxruct- 

ional programs are serving a particular target population. He calls, therefore, 
for an initiative in research. This is represented by the A/d/f/F interaction. 

Example: An organization engaged in development independently concludes that 

it would be useful to develop a certain process or product for instruction. 

This is represented by the B/PSj^ interaction. 

Example: Research is initiated for its own sake and pursued solely for the know- 

ledge which it produces. No findings have yet been incorporated either in 
development or operations. This is symbolized by the C/f, F, F relation in 
research. 

Example: Research initiated for its own sake yields the findings that certain 

organizational structures for large city school systems are always problematical or 
that a certain vitamin supplement administered between the ages of five and 
seven can prevent a form of mental retardation whose appearance cannot be detected 
until somewhat later. Neither one of these f?.ndings needs to pass through develop- 
ment. Each can be implemented directly in school operations (if so desired). 

This kind of relationship of research to practice is illustrated by the G/f, f/h/eOq 
interaction. ^ 

Example: Finally, consider an instance in the form of the linear flow or Guba- 

Clark model from research to development to implementation. Research on early 
childhood and cognitive growth, for example, uncovers a number of findings, some 
of which may suggest the development of processes and environments which can act- 
ively enhance such growth. Development efforts are consequently supported and 
carried through to completion. The availability of the resulting products and 
processes is made known to educational administrators who are thereby persuaded to 
incorporate the newly developed early childhood instructional programs, at either 
private or public expense, into the Nation's educational system. This kind of 
sequence is represented by the J/f, F, F, F/k/PS./l/EO. interaction^^. 



The Model as an Heuristic 

All of these representations in Figures II, III, and IV are fairly obvious 
and straightforward. The use of the model as an heuristic, however, profits from 
further explanation. One of these uses pertains directly to the problem of "change 
process" as applied to education. I have tried to structure the model (l) to 
illustrate that the incorporation of research findings into development is just 
as important as incorporating newly developed processes into operational settings 
( 2 ) and also to imply that it oft©ntim©s tray be ,1ust as difficult a proposition* 

The notion that there are obligations on both research and development to transfer 
their products" to other activities means that each must pay careful attention 
to the way in which its outputs are presented and, perhaps the very way in which 
the outputs are produced. In other words, the requirement that eventually there 
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be transfer or incorporation into another type of activity places constraints upon 
the professional behavior in each activity which cannot be ignored without com- 
promising later impact. 

This requirement is particularly true for development projects, but I think 
it is as true for research activities. A few concrete examples illustrate the 
point. A most simple one is the researcher who publishes perfectly valid findings 
in a sloppy or difficult format and thereby hinders the likelihood of their being 
incorporated ultimately into practice. The researcher who Inadvertently conceals 
or compromises his methodology or design encounters similar problems. 

Similarly, the final requirement for a development project is that it be us- 
able in operational settings. The ultimate desire to incorporate the developed 
process in school operations means that one of the performance specifications for 
development must always be the provision of procedures (teachers' manuals, train- 
ing procedures, etc.) for accomplishing the installation of the innovations. If 
the development is undertaken without reference to that fact (if, in short, the 
requirement for transfer is not built into the performance specifications), the 
developer may well have rendered his product unusable. Hence, for example, the 
desire to involve teachers and other practitioners in the development process stems 
from the need to have their expertise and experience continually represented. 

They carry embodied within them, in effect, the knowledge of many of the operational 
possibilities and constraints within which the finished process will operate 
There are other good reasons for involving teachers as well, not t\ least of 
them being the respectability lent to the project in the eyes of the practitioners 
at large by virtue of the meaningful presence of teachers in the effort. This 
last consideration is of no small importance in securing acceptance of the inn- 
ovation in the profession at large. The nature of its importance, however, 
should not be mistaken; the involvement must be meaningful and not merely window- 
dressing, for the respectability is lent by their presence only if their contri- 
butions are fully utilized and their real knowledge of what is or may be possible 
is thoroughly weighed in the development effort. 

One final point should be made about the model's portrayal of the possibility 
for transfers back and forth among research, development, and operations. Emphasis 
clearly needs to be given to the problem of information flow and the need for 
carefully considering techniques for installation of better knowledge and better 
processes into their intended settings. Only part of this is the direct respon- 
sibility of the researchers and the developers. Those with obligations for consid- 
ering the entire R&D system for improvement need also to direct their attention 
to these diffusion processes. 

A second feature of the model as an heuristic is the way it helps to clear 
the problem of distinguishing between basic and applied research in 
. The model clearly implies that basic research (studies generated 
independently in research for the sake of the findings alone) and applied research 
(research conducted to serve a particular need identified by people engaged in 
development of operations) differ from one another primarily in terms of the 
intent of the initiator^^. Thus the knowledge-orientation of the basic researcher 
is central to his activity. Applied research is also supported for the knowledge 
which results from it, but the initiator of the research knows to what instrumental 
use he is going to put the findings. By depicting both applied and basic research 
as similar kinds of activities, the model implies that jri anj of themselves they 
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iook very much alike. The procedures, the design, the sophistication must all be 
on par if either is to be valuable. What distinguishes the two from one another, 
if anything does, are the purposes for which they are initiated^^*^* 

A third feature of the model as a heuristic is its suggestion that decisions 
to initiate activities of each of the three types are made according to quite 
different criteria and perhaps by quite different people. The fundamental scien- 
tific character of research suggests that independent initiatives exercised there 
depend heavily on advice from the science community. Development projects, however, 
can also be independently initiated, but decisions to begin these kinds of act- 
ivities are subject to advice from both research and operations. With limited 
resources, deciding which needs to satisfy through development (for example, those 
independently generated by developers, compared to those stemming directly from 
school operations, compared to those growing out of research activities) becomes 
a particularly difficult problem. Finally, the kinds of lonely decisions required 
of school administrators 3t the operational level are made by people in the con- 
text of still different circumstances and institutions. By emphasizing the ess- 
entially different nature of the activities being undertaken, the model reminds 
the policy maker of the need to collect different kinds of data and statements of 
need when planning future activities. 

Finally, the frank attempt to represent each of the activities in terms of 
particular kinds of outputs may well turn out to be the most significant aspect 
of the model. It forces the viewer of the model to consider what the outputs of 
each activity are and to think about how the outputs of each activity are of use 
to one another. The outputs of research, for example, are knowledge. Some of the 
knowledge produced through research will find its way into development and into 
school operation. Are there ways of improving the output of research, making it 
more powerful, increase the likelihood of its being of use to instruction and ed- 
ucation? 

What about the outputs of development? They constitute, on the one hand, the 
validation of research and, on the other, the means by which the educational 
system can improve the manner in which it carries out its functions. How can 
development be improved, how can research be organized to be of greater use to 
development, and how can the educational system itself orient its organization to 
the recurring need for the installation of more powerful validated techniques? 

Finally, what happens to educational operations when they begin to view their 
responsibilities in terms of output? The contrast can perhaps be most sharply 
drawn by considering the implications of the notion of grading schools on the 
basis of their outputs rather than students on the basis of their performance. 

The existing practice of grading students assumes at bottom that the student is 
responsible for his learning and that his failure or success is a tribute to or 
a consequence of factors intrinsic to him. The idea of grading a school on the 
basis of its outputs assumes quite to the contrary that all students can learn 
and that the responsibility of the schools is to make that happen. (In medicine 
and law, for example, we judge success or failure of the system not so much by 
the patient's or client's end state as by the degree to which the doctors or 
lawyers skillfully utilized the most sophisticated practices in attempting to 
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serve the client. We certainly do not ’’grade" the patient or client; quite to 
the contrary, it is the professional services themselves which are assessed. An 
output orientation for school operations would cause the same reorientation of 
the direction of assessment in education.) If the schools themselves are judged 
in terms of the degree to which they are accomplishing their "production goals", 
increasingly they may come to orient their activities to assessing their own 
effectiveness, identifying the techniques and processes which need improvement 
and, as a consequence, calling with increasing sophistication for the kind of 
development activity and research support which will provide the basis for con- 
tinuing improvement. 



Summary 



I have, in this paper, presented an output-oriented model of educational 
research and development. I have tried to show that the outputs of research, 
development, and operating educational institutions are quite different, that 
performers of each of the three types of functions have important contributions 
to make in identifying proposed initiatives in their own sphere as well as all 
the others, and that these conditions create special demands upon the administrator 
of research. He must listen not only to science, but also to the behavioral tech- 
nologist and the professionals administering educational programs of all kinds. 

The model suggests the importance of adequate dissemination and diffusion mech- 
anisms among the different functions, the importance of the manner of performing 
the activities in each function in making transfer and feedback from one function 
to another possible, and the importance of collecting information from a broad 
range of input sources before making priority judgments. Finally, the discussion 
suggests that the notion of an output orientation in the educational system itself 
may well be the most significant procedural and managerial innovation we can 
think of because of the ways in which it may cause education professionals to 
identify deficiencies in service and seek out research and development which will 
continually contribute to their efforts to serve society well. 
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9. I am indebted to Robert Glazer for his .suggestion that somewhere (and here is 
as good a place as any) I make explicit the assumption that in education a 
breed of deve].opers and technicians exists, and that researchers, developers, 
and school administrators are presently skilled and courageous enough to 
identify and fulfill research and developments requirements (as Glazer put it 
to me, superhuman enough to know when and how to initiate development, how-to 
set up performance specifications, and possessing both the courage and political 
climate which will permit iterative experimentation and not just a "safe 
improvement"). Those assumptions are present not because I think conditions 

in the field fully warrant them, but because assuming that such conditions 
should be present seems to me to be useful. 

10. This example is, of course, clearly an ideal conceptualization. It is instruc- 
tive to keep the model in mind as one looks back over the past four years at 
the tremendous developing interest in the establishment of early childhood 
educational programs. While it is certainly difficult if not impossible to 
establish an individual cause for such a complex phenomenon, it is non^heless 
significant, I think, that Benjamin Bloom published his volume. Stability and ^ 
Change in Human Characteristics (John Wiley and Sons, Inc., New York, 1964)^ 

just at the time the Office of Economic Opportunity was beginning its planning 
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toward the development of programs to fight the war against poverty. The 
significant thing about Professor Bloom's volume, however, was that while 
his conclusions firmly underscored the importance of the early years in the 
development of cognitive skills, he also pointedly observed that there had 
been little actual experimentation designed to create environments to enhance 
such skill development. I do not think that it does violence to Dr. Bloom's 
argument to interpret it as a call for rigorous development efforts designed 
to produce environments and instructional programs which have the effect of 
enhancing human capabilities. The problem, however, is that which exists in 
all social domains. There are large numbers of children now whom we cannot^ 
afford to igno_:e, and the consequence has been the attempt to^create operating 
early childhood programs based on those convincing research findings without 
first having gone through a developmental stage. The result has-been a some- 
what marginal impact on the target population despite the clear implications 
of the research summarized by Prof. Bloom. 

I think it is also useful at this point to recall Professor^Reagan|s view 
that the social sciences present something of a different picture in regard 
to the distinction between basic and applied research. As he points out 
(Reagan, op. cit ., p. 1385), no matter how abstruse and abstract its pract- 
itioners attempt to become, social science research is inherently related to 
potential applications. 

The initiator and performer are, or course, not necessarily one and the same 
person or institution. The actual initiator of the research project might 
be a school man, a developer, a researcher, or a research administrator. The 
performer of that research effort may or may not have the same ultimate pur- 
pose as the initiator in mind as he undertakes the activity. Thus, for example, 
it would be perfectly possible for a research administrator to stimulate^ a 
series of research activities relating to reading which he views as applied 
research necessary for a development effort to build improved reading curri- 
cula, while the performer of that research sees it as a basic research effort 
in perception or the psychomotor skills associated with reading. 

This view, I believe, fits fairly well with one part of Harvey Brooks | analysis 
of the distinctions between basic and applied research (Brooks, op. cit. , p.^ 
1706) when he noted that "as definite categories, (the terms) basic and applied 
tend to be meaningless, but as positions on a scale within a given environment 
they probably do have some significance." The principal shortcoming with 
Brooks' analysis in my view is that it proceeds almost entirely on the presump- 
tion that the distinction can be resolved by approaching it^in some way from 
the researcher's point of view. Nly experience in the administration of 
research, as the model presented in the paper clearly indicated, leads me to 
believe that the researcher's view is only one of several which must be taken 
into account in attempting any analysis of the distinction criteria and 
alternative vantage points convinces me that Reagan's argument for abandoning 
the distinction within research is much more persuasive as is the suggestion 
that the critical categorization is that between research and development. 
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14. I thank Elvin Svanson who soma months SQOj in classic Socratic fashion during 
an axtandad intarchanga, provokad ma to clarify many of tha idaas prasantad 
in this papar. 
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Discussion 



The June 14th meeting opened wi-' a suminary of previous da y^s discuss^ 

by Dr. Moss, Conference Co-director. The cur ,,uxy was discussed and approved 

some additions and revision^. 

The paper by Dr. Gideonse was discussed and the following points and questions 

were representative group, including scientists, program operates, 

and others should make the decisiL concerning the allocation of educational 
research and development funds to various functions. These decisions have gre 
influence upon the direction of change in education; they act to mold the future. 

^ r Developmental projects may be initiated from any 

tion, representatives from development, research, and operation activit.es 

inpvit.sblv involvGcl in thG procGSS# 4.u^ 

3 There is a professional responsibility to disseminate accurately the 

result; of research, development and operation within ^he profession and to the_ 
Dublic. The form taken for dissemination, however, probably needs to be tailor 
tn qnecific audiences if it is to be meaningful and relevant, ihe develop 
ment process includes some dissemination and diffusion activities. Communica- 
tion with other developers and researchers may be the responsibili y 
qnpcial qrouD of translators or the developers themselves. ^ r 

4. The large majority of conferees favored treating diffusion of the pro- 
duct of the developmental process to operators as an activity f ® T*’® 

Lvelopmental process because of its magnitude and importance, and because of 
t^k^nds of complex problems and considerations it entai s. 

with operators should usually be accomplished by a special group °f gVg 

marketers who interpret relevant outcomes of research and development in accurate, 
TnderstandCe terms to special publics of operators, and who ha^e the respon- 
sibility of helping operators make more rational decisions a ou e p 

^ h One of the problems of marketing the developed products is that many 
school; ar^not now Lre of the outcomes and costs of their present programs. 
This information could provide the rational base for comparing the feasibility 
nf nn+pritial chanoes The developer, through the diffuser/marketer, has t 
professional responsibility of providing truthful data about his 
program outcomes (including costs) based upon formative and summative ev 1 
finriQ as a riGCGSsarv p3rt of tho cIgvgI opmGntsl procGSS# 

6 It is evident that the usefulness of the developmental process res s 
on the‘valLity and reliability of the criteria used to evaluate the product 
of the process. The profession has given insufficient attention to 
rivation of evaluative criteria and their measurement. 

A Dortion of the afternoon's discussion concerned question 2 of the con- 
ference guidelines (How should one go about planning or conducting educational 

development projects?). 
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1. A "theory model", described in detail in a project report by Drs, Maccia* 
was proposed as useful in the developmental process. A decision procedure derived 
in part from their theory model was offered as a strategy for planning educational 
development projects. 




O stands for a SPECIFICATION 



□ stands for a DECISION 



(a) U - A prerequisite to the application of the strategy is a problem, 
which seems to tentatively determine the universe of concern 
(e.g., goals, target group, program characteristics, interacting 
variables ) . 

(b) is the specification of all the critical elements (variables), 
or clusters of critical elements if necessary, in the universe. 

The critical elements are those with which developers should be 
concerned in considering the problem. There are also non-critical 
elements in the universe v'hich the developer will deliberately ig- 
nore because they are not germane to the problem. Reviews of re- 
search, practice, theory, and the use of consultants are techniques 
helpful in identifying the critical elements. 

(c) is the specification of all the relationships among critical 

dements , as these are determined by prior research, theory, 
practice, expertise, etc. Relationships among critical elements 
which are not known (cannot be specified) indicate needed research. 
In the event that one or more relationships are not known, several 
alternatives are available: Stop the process and redefine the 
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(h) 



the problem and universe; Consider alternative relationships and 
carry forward in subsequent steps the consequences of all alter- 
natives; Assume relationships and thus make these assumptions 
explicit for later diagnosis. 




(d) |D]^| represents the decision to select one or more critical elements 
as the dependent variables (goals) of the instructional process to 
be developed. 

is the specification of all of the relationships among critical 
elements which are related to the selected dependent (outcome) 
elements decided upon at|^. (S^ is thus a subset of and 

these relationships are constraints upon the dependent ^tcome 
elements. 

(f) (D 2 I is the decision as to the desired condition of the product of 

the developmental process. The critical elements of the develop- 
mental process will be manipulated (increased, decreased or changed 
in some other ways) to attain a desired condition. 

(g) 



is the specification of all of the relationships among the 
critical elements (from ) to optimize . Manipulation of 

these relationships (ie. inde pendent variables) will bring about 
the desired change (from M. ) in the value of the selected 
outcome elements (from ET ). They are the critical elements 
which the developmental product will seek to optimize. 



is the specification o f all of the other reactions (outcomes 
^ concomitant effects) which will occur when the relation- 
ships are optinuzed. relationships are subsets from those 

specified at 




^ is the go. no-go decision ; whether or not to proceed with the 
evelopment process. If too many undesirable effects of optimizing 
variables are anticipated by the specifications, then the 

ecision may be to change the EJ decision, to re-examine 
specifications, or even to alter the universe of concern. 



the 



2. A favorable outcome of the use of the strategy results in a decision 
concerning project goals (expected outcomes), the critical elements to be man- 
ipulated by the developmental product, the variables (critical elements) to 
control, and the anticipated side effects. 

3. The strategy model helps to specify the assumptions made (relationship 
among critical elements), the limitations on a project (as defined by the uni- 
verse), and the selection of variables to be controlled. If the product of the 
developmental process doesn't work, reference can be made to particular sequences 
in the strategy model (e.g assumptions) to detect the basis for failure. The 
strategy can also be useful in generating alternative procedures for conducting 
developmental projects by (a) the specification of ajternate relationships 
among critical elements, and (b) the specification of different sets of elements 
to be optimized. 
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4. Discussion began to focus upon the developmental steps which would 
follow the application of the above strategy. A useful distinction was re- 
cognized between the substantive development questions to be answered and the 
organizational-sociological decisions to be made. 

5. The former might include the need to answer questions like: How 

specifically should one's objectives be stated at the beginning of the project? 
Is this a possible function of project complexity? Can objectives be made 
content- free? How can the variables to be optimized be operationalized? In 
what form(s) should the product(s) be finalized? How instantaneously should 
feedback be secured during the development, and from whom should it be secured? 

6. The organizational-sociological questions involve how and by whom these 
questions should be answered, and the work of the project carried out. For ex- 
ample, what should be the relationship between formative evaluators and writers 
in the development process? How and when should consultants be employed? 

7. The major substantive steps following the design stage v;ere felt to 

be: (a) develop, (b) test (formative) diagnostically, (c) re-design and re- 

develop, '(d) retest (f orma tively ) , (e) repeat as needed, and (f) summative 
evaluation 



-73- 



Conferees’ Conclusions 



The conferees as a group agreed th9t no one model of the developmental 
process would be equally representative of all developmental projects. The 
complexity of the problem and its alms, and the uniqueness of those aims, are 
two variables which might Influence the choice of the most efficient process. 
But it is possible and desirable to develop a model of the developmental pro- 
cess at whatever the level of generality required. 

Individual conferees made the following concluding remarks: 

1. Every development project must Include a summatlve evaluation ( an 
estimate of an operational program's overall effectiveness) which compares the 
development product with the standard product. Schools need information on 
their own program's output and costs at the same level of precision as the 
developer's data so that they can make rational decisions concerning adoption; 
such data will tend to reduce uncertainty with respect to the dimension of the 
decision making process which is affected, whether this dimension is economic, 
political, or otherwise. 

2. Developers must clearly define their universe of concern, distinct 
from axiological considerations, but including a utility base. They must pro- 
vide for the disclosure of "side effects" of their products. An agency inde- 
pendent of the development resoonsibility is needed to show the rationality of 
using the new versus the old practice. 

3. Sociological considerations (the personalities and organizational ar- 
rangements involved) are a critical factor in determining the nature of the 
project goals, the actual outcomes, and the acceptability of those outcomes. 

4. There are two kinds of expected outcomes from developmental projects. 
The first are commonly accepted, standard goals which the project wishes to 
attain with maximum efficiency. The second kind of outcomes are new, unique 
educational ends. The latter introduce a great many value considerations 
into the developmental process, and make comparative (with existing pro- 
grams) evaluation difficult. 

5. The complexity of the problem selected for the developmental 
project influences the procedures used in the developmental process. It 
can, for instance, affect the decision concerning specificity of objectives 
which can or should be stated at the. outset of the project. Note also, 
that we can have small development projects within larger developmental 
projects . 








Conference Co-Directors’ Summary 



The three co-directors have reached the following tentative conclusions 
concerning (a) the substantive questions dealt with by the conferees, and (I) 
the conference procedures employed. 

1. Conferees were able tc reach substantial agreement upon the definition 
of developmental activities, and their relationship to research and to diffusion 
activities . 

2. There are essential similarities in the basic procedures and types of 
decisions required by research and by development activities,. Both are, after 
all, problem-solving activities and general problem-solving models, like Dewey's, 
can be applied with equal ease to both. 

3. On the other hand, when more specific procedures and decisions are con- 

sidered, research and development become quite distinguishable as the result of 
at least three differences in their respective products: (a) Development is 

committed to fabricating an instructional product to meet certain specifications, 
which implies reiteration of the design-development-test sequence; (b) develop- 
ment places its emphasis upon the fabrication of a product (treatment); (c) 
development requires that its product be operational; (d) development deals 
with educational systems, therefore it is necessary to consider both positive 
and negative outcomes of any changes made within the system; too often poten- 
tial negative outcomes are completely ignored. 

4. It appears evident that useful models can (and should) be specified for 
planning developmental projects. Differences in the size and complexity, and 
nature of the outcomes ( common, or unusual outcomes) of developmental projects, 
however, might dictate the need for adaptation in any one model that might be 
formulated , 

5. Models for planning developmental projects should probably incorporate 

provisions for considering two kinds of decisions: (a) Substantive, dealing with 

stages in the creation of the product, and (b) process, dealing with the utiliza- 
tion of human resources. 

6. Participants in the conference were not expected to evaluate preconceived 
ideas of the conference co-directors. Rather, the intent of the conference was 

to optimize creative inputs from the participants. In order to facilitate this 
objective (a) a relatively unstructured format was adopted, (b) the total number 
of participants was severely limited, and (c) neither the conference leader nor 
the co-directors felt compelled to direct the discussions or to censor points 
of view which were expressed. While the relatively unstructured nature of the 
conference may have been a little unsettling for some participants, it is the 
opinion of the co-directors that the procedures employed did serve the intended 
purposes reasonably well. 

7. Publication and dissemination of this report by the Minnesota Research 
Coordination Unit in Occupational Education represents the first step in a series 
of subsequent activities which might be taken as a result of the conference. 

Since many of the salient points presented in the papers and discussions are 
controversial, they are expected to provoke additional discussion among educators, 
researchers, and funders of educational development projects. Accordingly, the 
Coordination Unit's next steps will be to explore, in more detail, the adaptation 
and application of potential decision-making models to developmental projects 
which differ in terms of size, complexity, and outcomes. 






